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Abstract 



Differential privacy is a promising approach to privacy preserving data analysis with a well- 
developed theory for functions. Despite recent work on implementing systems that aim to 
. provide differential privacy, the problem of formally verifying that these systems have differen- 

tial privacy has not been adequately addressed. This paper presents the first results towards 
automated verification of source code for differentially private interactive systems. We develop 
a formal probabilistic automaton model of differential privacy for systems by adapting prior 
work on differential privacy for functions. The main technical result of the paper is a sound 
proof technique based on a form of probabilistic bisimulation relation for proving that a system 
modeled as a probabilistic automaton satisfies differential privacy. The novelty lies in the way 
' we track quantitative privacy leakage bounds using a relation family instead of a single relation. 

We illustrate the proof technique on a representative automaton motivated by PINQ, an im- 
plemented system that is intended to provide differential privacy. To make our proof technique 
easier to apply to realistic systems, we prove a form of refinement theorem and apply it to 
show that a refinement of the abstract PINQ automaton also satisfies our differential privacy 
definition. Finally, we begin the process of automating our proof technique by providing an 
algorithm for mechanically checking a restricted class of relations from the proof technique. 



"This work was partially supported by the U.S. Army Research Office contract on Perpetually Available and 
Secure Information Systems (DAAD 19-02- 1-0389) to Carnegie Mellon CyLab, the NSF Science and Technology 
Center TRUST, the NSF Cyber Trust grant "Privacy, Compliance and Information Risk in Complex Organizational 
Processes," and the AFOSR MURI "Collaborative Policies and Assured Information Sharing." 
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1 Introduction 



Differential Privacy. Differential privacy is a promising approach to privacy-preserving data 
analysis (see |Dwo08l ID wo 10] for surveys). This work is motivated by statistical data sets that 
contain personal information about a large number of individuals (e.g., census or health data). In 
such a scenario, a trusted party collects personal information from a representative sample with 
the goal of releasing statistics about the underlying population while simultaneously protecting 
the privacy of the individuals. In an interactive setting, an untrusted data examiner poses queries 
that the trusted party evaluates over the data set and appropriately modifies to protect privacy 
before sending the result to the examiner. Differential privacy formalizes this operation in terms of a 
probabilistic sanitization function that takes the data set as input. Differential privacy requires that 
the probability of producing an output should not change much irrespective of whether information 
about any particular individual is in the data set or not. The amount of change is measured in 
terms of a privacy leakage bound — a non- negative real number e, where a smaller e indicates a 
higher level of privacy. The insight here is that since only a limited amount of additional privacy 
risk is incurred by joining a data set, individuals may decide to join the data set if there are societal 
benefits from doing so (e.g., aiding cancer research). A consequence and strength of the definition is 
that the privacy guarantee holds irrespective of the auxiliary information and computational power 
available to an adversary. Previous work on algorithms for sanitization functions and the analysis 
of these algorithms in light of the trade-offs between privacy and utility (answering useful queries 
accurately without compromising privacy) has provided firm foundations for differential privacy 
(e.g. [DMNS0H1 iDwoOBl IMT071 INRS071 IbTrM IDwo081 [Dwo091 iGRSfM IDwolOl IDNPRlOj L 

In a different direction, these sanitization algorithms are being implemented for inclusion in data 
management systems. For example, pinq resembles a SQL database, but instead of providing the 
actual answer to SQL queries, it provides the output of a differentially private sanitization function 
operating on the actual answer |McS09j . Another such system, airavat, manages distributed data 
and performs MapReduce computations in a cloud computing environment while using differential 
privacy as a basis for declassifying data in a mandatory access control framework [RRS + l6"] . Both 
of these are interactive systems that use sanitization functions as a component: they interact with 
both the providers of sensitive data and untrusted data examiners, store the data, and perform 
computations on the data some of which apply sanitization functions. Even if we assume that 
these systems correctly implement the sanitization functions to give differential privacy, this is not 
sufficient to conclude that the guarantees of differential privacy apply to the system as a whole. 
For the differential privacy guarantee of functions to scale to the whole of the implemented system, 
the system must properly handle the sensitive data and never provide channels through which 
untrusted examiners can infer information about it without first sanitizing it to the degree dictated 
by the privacy error bound. 

Formal Methods for Differential Privacy. We work toward reconciling formal analysis tech- 
niques with the growing body of work on abstract frameworks or implemented systems that use 
differential privacy as a building block. While prior work in the area has provided a type system 
for proving that a non- interactive program is a differentially private sanitization function [RP10] . 
we know of no formal methods for proving that an interactive system using such functions has 
differential privacy. Applying formal methods to interactive systems ensures that these systems 
properly manage their data bases and interactions with untrusted users. 
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Formal verification that an interactive system provides privacy requires that the system be 
modeled in such a way that the correspondence between the system and model is evident and the 
model includes all relevant behavior of the system. Once formal verification is done on the model, 
one can assert with the confidence afforded by formal proofs that the system as implemented 
and modeled preserves privacy in addition to knowing that the algorithms implemented by the 
system preserve privacy. For formal verification to scale to large programs with complex models, 
the creation of the model and the verification of its privacy must be mechanized, preferably in a 
compositional manner. 

To this end, we present an automaton model for which the correspondence between the au- 
tomaton and the implementation of a system is so plainly evident that the automaton could be 
automatically extracted from source code as is done with model checking [CGPOOj . For this model, 
we introduce a form of compositional reasoning that allows us to separate the proof that a function 
gives differential privacy from the proof that the system correctly uses that function. Furthermore, 
we present a proof technique for such models that is amenable to mechanization and an algorithm 
that can be used to check that the proof technique is correctly applied to a model. 

Our effort can be likened to those efforts in the security community that involve the devel- 
opment of formal models for cryptographic protocols and the accompanying verification meth- 
ods [ST071 IBPW071 ICCK+08] . These works use stylized proofs with multiple levels of abstraction 



and compositionality to enable scaling mechanical checking of these proofs to the size of realistic 
systems. Making these proofs shorter or more readable for humans than their informal counterparts 
is not a goal. 



Contributions. We work with a special class of probabilistic I/O automata that allow us to 
model interactive systems in terms of states and probabilistic transitions between states. These 
automata provide us with the needed expressive power for modeling how data is stored in an inter- 
nal state of an implementation, and how it is updated through computations, some of which apply 
differentially private sanitization functions on data. In Section 13. 1\ we present this probabilistic 
automaton model and our differential privacy definition for probabilistic automata, which we call 
differential noninterference due to the similarities it has with the information flow property non- 
interference [GM82j . Indeed, when applied to interactive systems, both differential privacy and 
noninterference privacy aim at restricting information leakage about sensitive data by requiring 
that the system produces similar outputs for inputs that differ only in sensitive data. However, 
differential privacy allows for the degree of similarity to decrease as the inputs diverge, making it 
a more flexible requirement. 

As formal methods can only scale to large systems with compositional reasoning, in Section U 
we examine the ability to perform compositional reasoning with our formal model. We show that 
correctness proof of sanitization functions may be separated from the correctness proof of the 
system that uses them. 

Our main technical contribution, presented in Section [5l is a proof technique for establishing that 
a system has differential noninterference. Our technique allows the global property of differential 
noninterference to be proved from local information about transitions between states. This proof 
technique was inspired by the unwinding proof technique originally developed for proving that a 
system has noninterference |GM84] . 

Our unwinding technique is also similar to bisimulation-based proof techniques as both uses 
a notion of "similarity" of states with respect to their observable behavior. Unlike traditional 
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bisimulation relations for probabilistic automata, the unwinding relation is denned over the states 
of a single automaton with the intention of establishing the similarity of two states where one 
is obtainable from the other by the input of an additional data point. Moreover, the notion 
of similarity is approximate, which is in keeping with the definition of differential privacy. An 
unwinding proof involves finding a relation family indexed by the set of possible values of the 
privacy leakage bound e, rather than a single relation. This departure from traditional probabilistic 
bisimulations is needed to track the maximum privacy leakage tolerable from a given state in the 
execution. We prove the soundness of our proof technique in Theorem which roughly states 
that the existence of appropriate e-unwinding families for an automaton M implies that M has 
e-differential noninterference. 

As in other formal proof techniques of this nature, the real creativity in doing the proofs with 
our technique goes into defining the unwinding family. Unsurprisingly, the rest consists of repeated, 
routine applications of basic arguments showing that the defined relation between states is preserved 
by transitions of the system. In Section [61 this quality enables us to develop an algorithm to check 
whether a given relation family is an unwinding family, thereby automating proofs for differential 
noninterference modulo the definition of the relations. We prove that the algorithm soundly runs 
in polynomial time: it will only return true if the automaton has e-differential noninterference 
(Theorems H] and [5]) . 

To motivate our work, we start by presenting a system similar to pinq. We refer to the example 
system throughout our paper as we model it in our formalism and use our unwinding technique and 
algorithm to verify that it has differential noninterference. As PINQ may be configured to use any 
set of sanitization functions, we present an automaton M ex i that is parametric in the sanitization 
functions that it uses. We show two methods for proving differential noninterference for any correct 
instantiation of M ex \ with differentially private sanitization functions: by using the composition 
method presented in Section [H and by using our unwinding verification algorithm. This second 
method illustrates the applicability of our algorithm in proving differential noninterference for 
interesting automata. 

Along the way, we find interactions between a bounded memory model and differential privacy 
of interest beyond formal verification. In particular, we find the inability to store an unbounded 
number of data points results in doubling the privacy leakage. 

We finish with Section [7] covering related work and Section [8] presenting future work and con- 
clusions. 

2 Background and Motivation 
2.1 Differential Privacy 

Differential privacy formalizes the idea that a private process should not reveal too much information 
about a single person. A data point represents all the information collected about an individual 
(or other entity that must be protected). A multiset (bag) of data points forms a data set. A 
sanitization function k processes the data set and returns a result to the untrusted data examiner 
that should probabilistically not change whether or not a single data point is in the data set. 
Dwork [Dwo06j states differential privacy as follows: 

Definition 1 (Differential Privacy). A randomized function k has e-differential privacy iff for all 
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data sets B\ and B2 differing on at most one element, and for all S C range(K), 

Pr[«(J3i) e S}< exp(e) * Pt[k(B 2 ) G S] 

Formally, multisets B\ and B2 differ on at most one element iff either B\ = B2 or there exists 
d such that B\ U {d} = B2 or B2 U {rf} = B\. Note that the above definition is well-defined only if 
range(K) is countable. 

Differential privacy has many pleasing properties. For example, if B\ and B2 differ by n dat- 
apoints instead of just one, then the probabilities of k{B\) and k[B2) being in a set S will be 
within a factor of exp(n * e) of one another [MT071 Corollary 4]. Furthermore, a function that 
sequentially applies n functions each with e-differential privacy and provides all of their outputs is 
an (n * e)-differentially private function [MT071 Corollary 5]. 

Privacy Mechanisms. As shown in the original work on differential privacy, given a statistic 
/ that can be computed of the data sets Bi, one can construct a sanitization function Kf from / 
by having Kf add noise to the value of f(B{) where the noise is drawn from a Laplace distribu- 
tion [DMNS06]. This is an example of a privacy mechanism, a scheme for converting a statistic 
into a sanitization function with differential privacy. 

Systems in practice would implement a sanitization function such as Kf as a program. As actual 
computers have only a bounded amount of memory, the program computing Kf must only use a 
bounded amount of memory. However, many sanitization functions proposed in the differential 
privacy literature, including all sanitization functions constructed using the Laplace privacy mech- 
anism, use randomly drawn real numbers, which requires an uncountably infinite number of states. 
While such functions can be approximated using a finite number of states (e.g., by using floating 
point numbers), it is unclear whether the proofs that these functions have differential privacy carry 
over to their approximations. 

As we are interested in formally proving that finite systems provide differential privacy, we 
limit ourselves to privacy mechanisms that operate over only a finite number of values. One such 
mechanism is the Truncated Geometric Mechanism of Ghosh et al. |GRS09j . which uses noise drawn 
from a bounded, discrete version of the Laplace distribution. As we are interested in applying formal 
methods to systems using such mechanisms, we provide an implementation of this mechanism that 
runs in expected constant time and proofs about it in Appendix [Al 

2.2 Motivating Example System 

To further motivate and illustrate our work, we provide an example of an interactive system that 
uses sanitization functions. Throughout the remainder of this paper, we apply the various formal 
methods we develop to prove that it preserves privacy. The system manages data points entered by 
data providers and processes requests of data examiners for information by receiving queries and 
answering them after sanitizing the answer computed over the data set. The system must apply 
the sanitization functions to the data set and interact with the data examiner in a manner that 
does not compromise privacy. 

Possible source code for one such system is shown in Figure HJ To be concrete, suppose that the 
data points are integers and the system handles only two queries. The first produces the output 
of the sanitization function COUNT, which provides the number of data points currently in the 
data base. The second produces the output of SUM, which provides their sum. In both cases, 
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01 dPts:= emptyArray (t) ; 

02 mimPts := emptyArray (t) ; 

03 for(j :=0; j<t; 

04 dPts[j]:= emptyArray (maxPts) ; 

05 mimPts[j] := 0; 

06 curSlot:=0; 

07 while (1) 

08 y : =input () ; 

09 if (datapoint(y) ) 

10 if (numPts [curSlot] <maxPts) 

11 dPts [curSlot] [numPts[curSlot]] :=y; 

12 numPts [curSlot] ++; 

13 else 

14 k:=get_sanitization_funct(y) ; 

15 res :=k. compute (dPts) ; 

16 print (res) ; 

17 curSlot := (curSlot + 1) mod t; 

18 delete dPts [curSlot] ; 

19 dPts [curSlot] := emptyArray (maxPts) ; 

20 numPts [curSlot] := 



Figure 1: Program that tracks data point usage to ensure differential noninterference 

the sanitization functions use the Truncated Geometric Mechanism to preserve privacy jGRS09j . 
(Appendix IA.3I provides source code for count and SUM.) 

Intuitively, the program in Figure Q] keeps an array of t arrays of data points and a variable 
curSlot, whose value indicates a (current) slot in the array. If the input is a data point, that data 
point is added to the array indexed by curSlot unless that array is full, in which case the data 
point is ignored. 

If the input is a query, then the query requested by the input is computed on the union of all 
the data points collected from all the arrays. Line 15 uses either the implementation of COUNT or 
SUM to compute the system's response to the query y where Line 14 selects the correct function. 
Furthermore, the index curSlot to one of these arrays is cyclically shifted and the array to which it 
now points is replaced with an empty array. Since there are only t slots, this means that each array 
will only last for t queries before being deleted. (If t = 0, we take the program to have an array 
dPts of length 0, in which case it never stores any data points.) Since each query has e-differential 
privacy, this ensures that each data point will only be involved in t * e worth of queries. 

Verification. The goal of our work is to formally verify that systems like this one preserve the 
privacy of their users. In addition to showing that the sanitization functions count and SUM 
have differential privacy (a subject of previous work |GRS09j ). we study how the system leaks 
information about the data points in ways other than through the outputs from these functions. 
Indeed, one might expect from the sequential result for differential privacy discussed above |MT07t 
Corollary 5], that the system would provide (t * e)-differential privacy. However, due to how the 
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system manages data points, it actually only provides (2t * e)-differential privacy as we show later. 

Had our goal only been to formally verify the implementations of the sanitization functions 
COUNT and SUM, it would suffice to use a simple formal model such as that of probabilistic finite- 
state automata with no interaction and use a suitable algorithmic technique to verify differential 
privacy, which research on Markov chains provides. (We provide further details in Section [4.11 ) 

However, to verify differential privacy for interactive systems that use privacy mechanism as a 
building block as the above system does, we need a more expressive formal model that models the 
interaction of the data examiner with the system and the addition of data points to the system 
over time. The next section provides such a model. 

3 Modeling Interaction for Formal Verification 

In this section, we present the basics of the formal framework we use in modeling interactive systems 
and show how we can model the example system of Section 12.21 using this formalism. Specifically, 
in Sections 13.11 and 13.21 we introduce a special class of probabilistic I/O automata and present 
our definition of differential privacy for this class of probabilistic I/O automata. In Section [3.31 we 
model the program of Figured] as a probabilistic I/O automaton. 

3.1 Automata 

We use a simplified version of probabilistic I/O automata (cf. |LSV07j ). We define an automaton 
in terms of a probabilistic labeled transition system (plts). 

Definition 2. A probabilistic labeled transition system ^plts,) is a tuple L = (S, I, O, — >) where S 
is a countable set of states; I and O are countable and pairwise disjoint sets of actions, referred to 
as input and output actions respectively; and ->CSx(/UO)x Disc(S') represents the possible 
transitions where Disc(S') is the set of discrete probability measures over S. 

We use A for I Li O. We partition the input set / into D, the set of data points, and Q, the set 
of queries. We also partition the output set O into R, the set of responses to the data examiner's 
queries and H, the set of outputs that are hidden from (not observable to) the data examiner. 
Note that H includes outputs to the data provider. We let E range over all actions to which the 
examiner has direct access: E = Q U R. When only one automaton is under consideration, we 
denote a transition (s,a, fi) G — > by s4/i. 

Henceforth, we require that pltss satisfy the following conditions: 

• Transition determinism: For every state s £ S and action a £ A, there is at most one 
/i G Disc(S') such that s A fi. 

• Output determinism: For every state s G S, output o G O, action a £ A, and [i G Disc(5'), if 
s fi and s A //, then a = o and // = fi. 

• Quasi-input enabling: For every state s G S, inputs i\ and i2 in /, and /ii G Disc(S), if s-V //i, 
then there exists fi2 such that s /i2 • 

Output determinism and quasi-input enabling means that the state space may be partitioned into 
two parts: states that accept all of the inputs and states that produce exactly one output. We 
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require that each output producing state produces only one output since the choice of output 

should be made by the plts to avoid nondeterminism that might be resolved in a way that leaks 

information about the data set. Owing to transition determinism, we will often write s4/i without 

explicitly quantifying fj,. 

We define an extended transition relation => that describes how a plts may perform a sequence 

of actions where some of the output actions are hidden from the data examiner. In particular, 

the hidden outputs in H model unobservable internal actions irrelevant to privacy. To define 

let a state that produces an output from H be called H -enabled and one that does not be called 

H -disabled. By output determinism, if-enabled states may only transition under an action in H 

and, thus, cannot have transitions on actions from R U Q U D. To skip over such states and focus 

on if-disabled states, which are more interesting from a verification point of view, we define =^ to 

show to which if -disabled states the system may transition while performing any finite number of 

hidden actions. We define s v so that v{s') is the probability of reaching the f/-disabled state 

s' from the state s where a is the action performed from state s. Note that v is not a distribution 

over the set S of states since the automaton might execute an infinite sequence of If -enabled states 

never reaching an H -disabled state. We let v be a distribution over S± = S U {_!_} where _L ^ S 

represents nontermination and f(_L) = 1 — X^ses^( s )- Note that for no a, fi, or v does _L A fi or 
i a 

_L=^-ZA 

A plts L combined with a state s defines a probabilistic I/O automaton (L,s). This state is 
thought of as the initial state of the automaton or the current state of the plts. We define a trace 
to be a sequence of actions from A* U A u . Given such an automaton M, we define [M] to be a 
function from input sequences to the random variable over traces that describes how the automaton 
M behaves under the inputs i. We let wJ-E denote the random variable over sequences of 
actions observable to the data examiner obtained by projecting only the actions in E from the trace 
returned by random variable [[M](i). 

To deal with nontermination, we note that the examiner can only observe finite prefixes of 
any nonterminating trace. When the examiner sees the finite prefixes of a trace, he must consider 
all traces of the system with the observed prefix as possible. (The set of these traces has been 
called a cone — see e.g. |LSV07| .) Since the examiner may only see actions in E, these sets are 
in one-to-one correspondence with E* . Thus, the examiner observing some event is not modeled 
as the probability of the system producing a trace in some set, but rather with the probability of 
a system producing a prefix of trace in some set. That is, rather than using Pr[|jA/]](*)J-E £ S] 
for S C E* U E u , we need Pr[Lp-f](i)j£ □ S] for S C E* where □ is the super-sequence-equal 
operator raised to work over sets in the following manner: e □ S iff there exists e f € 5 such that 
e □ e" where e <E E* U E w and S C E*. 

In Appendix |Bl we formalize these concepts and show how to calculate these probabilities from 
the transitions of the automaton. 

3.2 Differential Noninterference 

Often the data set of a differentially private system is loaded over time and may change between 
queries. Such changes in the data set are not explicitly modeled by the definition of differential 
privacy, but one could conceive of modeling such changes by having data points be time-indexed 
sequences of data. Nevertheless, for formal verification, we require an explicit model of data set 
mutation. Thus, we present a version of differential privacy defined in terms of the behavior of an 
automaton that accepts both queries and data points over time. 
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Definition 3 (Differential Noninterference). An automaton M has e- differential noninterference 
if for all input sequences %\ and i<i in I* differing on at most one data point, and for all S C E* , 

Pr[L[Jtf](ti)J B □ S] < exp(e) * Pv[[{Mj(z 2 )\ E □ S] 

where we say two input sequences differ by one data point if one of the sequences may be constructed 
from the other by inserting a single data point anywhere in it. 

By restricting the traces of M to only those elements of E = QL)R, we limit traces to only those 
actions accessible to the untrusted data examiner. The definition requires that any subset of such 
traces be almost equally probable under the input sequences %\ and i 2 , which differ by at most one 
data point. Note that like the original form of differential privacy, we do not model the adversary 
explicitly but rather consider the behavior of the automaton over all possible input sequences the 
adversary could supply. 

In Appendix [Cj we give full definitions for sequence differencing and prove results showing 
that our adaptation of differential privacy preserves pleasing properties of the original. One such 
property is a composition result (Proposition \13\i : the privacy leakage bound for a system whose 
inputs differ on at most n data points is n * e where e is the leakage bound for the system if its 
inputs differ on one data point. 

3.3 Example: Automaton Model for Program of Figure [1] 

To eventually prove that the program of Figure [1] has (2t * e)-differential noninterference, we first 
give an automaton model of the program, called M ex i(K). Note that the model we give here is 
parametric in the set of sanitization functions; it applies not only to the program of Figure [TJ which 
assumes K = {count, sum} but to any other instance of the same program that uses a possibly 
different set of sanitization functions (modeled by the parameter K). We define below the state 
space S and transition relation — >, which determine L ex i(K) = {S,I,0,—>) for every set K of 
sanitization functions. Using an initial state sq, we get the automaton M ex \(K) = (L ex i(K), sq) . 

States. Each state of the automaton can be viewed as a particular valuation of the variables in the 
program allowed by its type. We model the array dPts as a i-tuple of multisets of data points. We 
model numPts as a t-tuple of integers ranging from to v where v is the value held by the constant 
maxPts. We model the index curSlot as an integer c ranging from to t — 1, which selects one of the 
multisets of the t-tuple. The variable y stores the most recent input. The variable res keeps track of 
which output from O is about to be produced and the sanitization function is stored in k. The state 
must also keep track of a program counter pc, which ranges over the program line numbers from 01 
to 20. Thus, the set of states S is {01, . . . , 20} x (bag(D))* x {0, . . . , v} 1 x {0, . . . , t - 1} x I x O x K 
where bag(D) is the set of all multisets with elements from D and K is the set of sanitization 
functions. 

Actions. We model the input command in the source code with the input action set / of our 
automaton: for each possible value that input can return there is an input action in / corresponding 
to that value. Inputs in the code can be either queries or data points, which is modeled by the 
partition of the set / into the sets Q for queries and D for data points. We model the print 
command in the source code with the observable outputs R (responses) of our automaton. For 
each possible value that can be printed we have an output action in R. We model all other 
commands by internal (hidden) actions. 
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Transitions. We list below only those transitions that are interesting for our purposes. That 
is, transitions on actions from the sets I and R, and transitions on hidden actions that represent 
internal computation such as choosing of an appropriate sanitization function for a given query 
and computation of the result using that function. We use the symbol r for hidden actions. We 
also use Dirac distributions: let Dirac(s) be the distribution such that Pr[Dirac(s)=s] = 1 and 
Pr[Dirac(s)=s'] = for all s' 7^ s. Given a query q in Q, we let K q be the sanitization function that 
answers that query. Some key transitions are: 

Input (08, B,ft, c, y, r, k) A Dirac((09, B, n, c, i, r, k)) 

Choose Function (14, B, ft, c, y, r, k) A Dirac((15, B, ft, c, y, r, K y )) 

Compute Function (15, (Bo, . . . , Bt-i),ft), c, y, r, k) -A /1 where 

t-1 

M(16, (B , . . . , ft, c, y, r', k)) = Pr[fc( [+J B e ) = r'] 

e=o 

using tt) for multiset union and fJ.(s') = for states not of that form, and 
Output Result (16, B, n, c, y, r, k) A Dirac((17, B, n, c, y, r, k)) 

The third transition above is a probabilistic transition that represents the internal computation 
of a sanitization function k on the union of the multisets Bq, . . . , B t _\. The effect of the transition 
is to update the value of the pc from 15 to 16 and to update the result to be output from r to a new 
value r' such that the probability of ending up in state (16, (B\, . . . , Bt),c, re, y, r' , k) as a result of 
the transition is Pr[/c([+J^ =1 Bg) = r']. 

From these transitions, we can calculate the extended transitions for each of the three types of 
//-disabled states: 

Drop (08, B, n, c, y, r, k) =4 Dirac((08, B, n, c, d, r, k)) when re c of ft is v; 

Add (08, B, re, c, y, r, k) 4> Dirac((08, B' , ft', c, d, r, k)) when n c of ft is less than v and B' and ft' 
are such that B' c = B c & {d}, n' c = n c + 1, and for all d 7^ c, B' c , = B c i and n' c , = n c f, 

Answer Query (08, (Bq, . . . , Bt-x),ft, c, y, r, k) 4> v where 

t-i 

v((16,(B ,...,B t - 1 ),ft,c,q,r',K q ))=-Pr[k([t)B e )=r'] 

and v(s') = for states not of that form; and 

Delete Old Data (16, B, it, c, y, r, k) =4> Dirac((08, B' , ft, c, y, r, k)) 

where we have B' c+1 mod t = {}, n' c+1 mod t = 0, and for all c" 7^ c + 1 mod t, B' c „ = B c » 
and n' c „ = n c » using for the empty multiset. 

The third extended transition above represents a sequence of transitions that starts with the 
input of a query q. The input of the query is followed by transitions on hidden actions that model 
the computation of the answer to the query where some of these hidden steps are probabilistic. 
The resulting state has the property that K q has been chosen as the sanitization function and that 



10 



pc = 16, which implies that the resulting state is ii-disabled and the automaton is ready to perform 
an observable output by outputing the answer to the query. 

The state space S and transition relation — > determines the plts L ex \(K) = (S,I,0,—t) for 
every set K of differentially private functions. Using the initial state sq = (1, O ,0*, 1, yo, ro, ko), 
we get the automaton M ex i(K) = (L ex i(K) , so) ■ (The initial values yo, ro, ko do not matter since 
they will be replaced before being used.) 

Verification of Differential Privacy and Bounded Memory. The remainder of this paper 
develops the proof techniques needed to formally verify that models such as the one shown above 
has differential noninterference. In particular, in the next section, we describe a composition result 
that allows to separately consider whether the sanitization functions in K have differential privacy 
and whether M ex i properly uses them. In Section [5j we present a proof technique using unwinding 
families for showing that for all sets K of sanitization functions with e-differential privacy, the 
automaton M ex i(K) has (2t * e)-differential noninterference. Lastly, Section [61 provides a proof- 
checking algorithm that ensures our unwinding technique is properly used. These methods together 
allow for a compositional and mechanically checked formal proof of differential noninterference. 

Given that the system modeled above uses e-differentially private functions t times, one might be 
surprised that we prove that it has (2t * e)-differential noninterference rather than (t * e)-differential 
noninterference. This extra leakage comes from dealing with the bounded memory of actual com- 
puters. In particular, each array in dPts is limited to a length of maxPts. The program keeps 
track of the current number of data points stored in each slot with the array numPts. If the current 
slot has reached maxPts data points, the program drops any incoming data points until curSlot 
advances. 

This dropping of data points introduces extra privacy leakage. A single data point can have 
two effects: it is both included in calculations and can cause the system to drop future data points 
and exclude from calculations. Thus, the system has only (2t * e)-differential noninterference. In 
many scenarios, the possibility of running out of memory for storing data points is unrealistic. If 
the number of data points can never reach the memory bound, then under this assumption, one 
can show that system has (t * e)-differential noninterference. 

It may be tempting to use a linked list for each slot and keep track of how many total data 
points are stored in all the slots combined. Then, the program could drop data points only when all 
the memory is exhausted instead of just the current slot's allocation. However, this change would 
allow a single data point stored in one slot to affect which data points are dropped from other 
slots in the future. Thus, a single data point may have an unbounded effect on future computation 
preventing such a program from satisfying differential noninterference for any privacy bound. 

4 Decomposing Verification 

Recall the example system presented in Section I2.2L The source code in Figure Q] is written para- 
metrically in the set of sanitization functions (Lines 14 and 15). The model M ex i(K) of the system 
given in Section 13.31 is parametrized over the set of sanitization functions K where the computa- 
tion of a sanitization function from K is idealized as a single transition in the transition system of 
M ex i {K) . We will call such models in which computation of functions are abstracted as a single step 
idealized models. In reality, any function in the set K would be implemented by a subroutine that 
can be modeled by an automaton and an implementation model could be obtained from an ideal- 
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ized model by replacing each idealized transition for a sanitization function with its corresponding 
subroutine automaton. 

In this section, we first provide an algorithm for checking that such subroutine automata mod- 
eling sanitization functions have differential privacy. Second, we show how to use the proof that a 
subroutine has differential privacy to simplify the task of proving that an interactive system using 
that function has differential noninterference. That is, we show how we support compositional 
reasoning by separating the verification of a sanitization function from the verification of a system 
that uses the function. 

4.1 Mechanized Verification of Differential Privacy 

Previous work has provided a method of formally verifying that a sanitization function has dif- 
ferential privacy jRPlOj . Their method operates over a special language to enable type-checking. 
Below we provide an alternative using automata to model the function. 

In particular, we model a subroutine implementing a sanitization function k operating on the 
database B using an I/O automaton Mk t B- As k performs no I/O, the model M^b has an empty 
set of inputs and only one output h, a hidden action. The initial state of Mf, b represents the start 
of the computation k operating on the argument B. For each output r in the range of k, M^^b has 
a terminal state £(r) with no outgoing transitions corresponding to returning the value r. Since k 

is a function, sq=^u must be a distribution over these terminal states with f(-L) = and u(s) = 
for all states not corresponding to an output. 

A function k has e-differential privacy only if Bi an d b 2 induces sufficiently close distri- 
butions over related terminal states for all data bases B\ and B2 differing by at most one data point. 
In particular, for all r in the range of k, fi(£i(r)) < exp(e) * 1^2(^2(1")) where V{ is the distribution 
over terminal states induced by the automaton M\. b 4 and £j is the mapping from the range of k to 
terminal states for Mj^Bi- 

Thus, mechanically checking if a function k has differential privacy reduces to constructing the 
appropriate models M^^Ba computing the distributions V{ for each of them, and comparing them 
as needed. As we are only concerned with systems that can actually be implemented, only a finite 
number of models and comparisons are needed. The construction of the models may be done using 
known techniques from model checking (see, e.g., |CGP00j ). The most complex step is computing 
the distributions V{. 

Fortunately, each of these automaton models Mj. g. may be converted to an absorbing Markov 
chain, a model of random behavior leading to one of a fixed set of absorbing states each representing 
a different outcome. Under this conversion, the probability of the Markov chain leading to a 
particular absorbing state corresponds to the distribution over terminal states of M). # r This 
conversion starts with finding the set S' of all if-disabled states reachable from sq by using hidden 
actions. For this task, we may view the transition system as a directed graph G where the nodes 

are states. If s — > /i and fi(s') > for some hidden action h, then we add an edge from s to s' 
labeled with fi(s') to G. (Recall that s will never transition under more than one such hidden 
action due to the transition-determinism axiom.) A depth-first search may then find those states 
reachable from s in G. Second, we remove all states from G that are not reachable from s. Third, 
we convert all states in G that are reachable from s that do not reach any i?-disabled states to a 
single state s±, which we treat as an i7-disabled state. We can do this with a reachability analysis 
for each state to every if-disabled state. Forth, we add a self-loop labeled with probability 1 from 
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every //-disabled state (including s±) to itself. The resulting graph corresponds to an absorbing 
Markov chain where the //-disabled states (including s±) are the absorbing states. 

To compute the absorbing probabilities of the //-disabled states, we use the standard method 
as presented in |GS97j . First, we represent the chain using a transition matrix P in canonical form. 
That is, we renumber the states so that the non-absorbing, or transient, states come first in P. 
In our case, these are the //-enabled states. Let t be the number of transient states and r be the 
number of absorbing states. We may view P as having the following form: 



P = 



f Q 


R 





I 



where Q is a t-by-t matrix, R is a non-zero t-by-r matrix, I is a r-by-r identity matrix, and is 
a r-by-t zero matrix. Here, Q, R, and I capture, the probabilities for, respectively, moving from 
a transient state to a transient state, moving from a transient state to an absorbing state, and 
moving from an absorbing state to an absorbing state. Second, from P, we compute fundamental 
matrix N = (I — Q) _1 . Third, we compute A = NR. The entry au of A is the probability of 
the chain ending in (being absorbed by) the state numbered j when started in the state i. Thus, 
we may set v{s ! ) = aij where i is the number of the initial state and j is the number of the state 
s' . We refer the reader to |GS97j for the correctness of this algorithm for computing the absorbing 
probabilities. 

Algorithm closure(M, s, a). The above algorithm may be generalized to compute v for a state 
s and an action a where s => v. The generalization replaces initial state with s and constructs 
the terminal absorbing states from the //-disabled states reachable from s. Let closure(M, s, a) 
denote the generalized algorithm used this way to compute v such that s=!> v. 

As for the runtime of closure, note that the first step of constructing of the graph G runs in 
0(|5|) where S is the state space of Mfc^. Converting G to use s± takes 0(|»S| 2 ). Every other 
step of the conversion process runs in 0(|5|). The matrix operations used to compute the matrix 
A can all be done in 0(|,S| 3 ) as t < \S\ and r < \S\. Thus, it runs in 0(|5| 3 ) time. Using closure 
for computing each z/j, we may check if k has differential privacy in 0(m + \B\ * \D\ * IS"! 3 ) where 
m is the time required to compute all the models and B is the set of all databases Z?j. 

4.2 Implementation and Composition 

The ability to verify that a subroutine provides differential privacy aids the verification that a system 
using that subroutine has differential noninterference. In particular, this section shows that the 
verification of differential noninterference may assume that the subroutine provides a differentially 
private distribution over return values in a single idealized transition, without modeling the internal 
transitions of the subroutine. Doing the verification based on such an idealized model is more 
manageable than doing it based on a model that includes the details about the implementation of 
the subroutine. 

Implementing a Transition with an Automaton. We now define what it means in our model 
for a single step transition on a hidden action to be implemented by an automaton with a series 
of hidden transitions. We base our notion of implementation on hidden transitions since it is 
sufficiently general for our purposes — we do not concern ourselves with the general question of 
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preserving all kinds of observable behavior through implementation but rather the more restricted 
question of preserving the resulting distribution over computed values. 

A single internal transition of an automaton Mi may result in a distribution over next states 
that corresponds to the distribution over terminal states induced by many internal transitions in 

another automaton M2. To formalize this, let be a state of the automaton Mi such that s^ \ x 
for some hidden action h) of M\. Let 1 be an injection from Supp(/^) to the state space of some 
other automaton M2 such that every state in the image of 1 is disabled for every action (i.e., they 
are terminal states). We say that the automaton M2 implements the transition of s' under 1 if 
for all s G Supp(/U^), = J2h^H + ^([]){h, t(s)) where H2 is the hidden action set of M2 and 

i?2 is the set of non-empty finite sequences using elements from H^- That is, M2 implements the 
transition of s' under 1 if the distribution over the terminal states that M2 reaches is isomorphic 
to fv under 1. 

Subroutine Composition. Subroutine composition can be viewed as replacing a single step 
transition in an idealized model with its automaton implementation where such repeated replace- 
ments can be used to derive an implementation model from the idealized model. 

Let Mi [s * , M2 , l] denote the automaton that results from replacing an internal transition from 
the state s^ of M\ with the subroutine M2 with the injection 1 providing how to return from the 
subroutine. Formally, given M 1 = ((Si, Qi W Di, Ri W Hi, -> x ), *>?}, M 2 = ((S 2 ,®,H 2 ,^ 2 ),s%), 

s^ G Si such that s^ \ x ^ for some hidden action h) G Hi and \$ where s^ is the unique state that 
enables h\ s^ ^ Supp(//T), and 1 : Supp(//t) — > S2 such that every state in its image is disabled for 
all actions, let Mi[s\ M 2 , t] denote the automaton M 3 = ((Si ttl S 2 , h, Ri W H x l±) H 2 W {h*}, ->- 3 ), s?) 
where ttl is disjoint union and —7-3 is defined as follows: 

• si ->3 /j, if si G Si, si 7^ s\ and s\ -4i ji; 

• S2 -4 3 /i if s 2 G 5*2 and s 2 -*- 2 A 4 ! 

• A3 Dirac^); and 

• i(si) A3 Dirac(si) for all si G Supp(//t). 

The special hidden action h$ in the definition of M3 above is used to mark the entry and exits 
points of the subroutine represented by M 2 . This extra action is used to correctly "hook up" M2 
with Mi to obtain M3. 

The lemma below states that if some internal transition of an automaton Mi (for example, a step 
corresponding to calling a sanitization function in a differentially noninterference system) is replaced 
by an automaton M2 (for example, multiple steps corresponding to a subroutine implementing the 
sanitization function), then the observable behavior of the resulting automaton is identical to that 
of Mi. 

Theorem 1 (Subroutine Composition). For all automata M x and M 2 , states s\ and injections 1 
such that M 2 implements the transition of s^ under 1, for all i in I* , and e in E* , 

Pr[ [Mi(i)\ E □ e] = Pr[ [Mi[s\ M 2 , t](i)\ E 3 e] 
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In Appendix [Dl we prove this by way of two lemmas. 

A corollary is that if an idealized model has differential noninterference then a implementation 
model formed by replacing its internal transitions with subroutine automata also has differential 
noninterference. 

4.3 Example: Decomposing Verification 

Suppose that M ex 2 is the automaton obtained from M ex i ({count, sum}) by replacing the transi- 
tions that represent the computations of the functions COUNT and SUM with subroutine automata 
-^count,^ arid M SUMi # 4 . That is, M ex 2 is the code shown in Figure CD with the implementations of 
COUNT and SUM in-lined. We may apply the composition theorem repeatedly for each replacement 
of a single transition in M ex i({couNT, sum}) with a subroutine automaton in M eX 2. Such repeated 
compositions reduces the problem of verifying differential noninterference for M ex 2 to two smaller 
problems: First, we must show that the automata M C0lJNTt Bi and M SUMi s i implement with a se- 
ries of internal transitions the transitions corresponding to the functions count and SUM found 
in M ex i ({count, sum}) as described in our formal definition of implementation. Second, we must 
show that the idealized model M ex i({couNT, sum}) has the differential noninterference. 

The first problem can be solved using closure, which establishes that the automaton correctly 
implement count and SUM. As COUNT and SUM has differential privacy (proofs provided in Ap- 
pendix |A|) . we may conclude that these subroutine automata have differentially private distributions 
over their terminal states0 The next two sections deal with solving the second problem. 

While COUNT and SUM are simple sanitization functions, the above approach generalizes to 
more complex sanitization functions: As long as the function can be modeled as a series of internal 
transitions that ends in states corresponding to its return values, our approach will apply. While 
most of the algorithms previously published use unbounded state spaces, we believe our approach 
can handle bounded versions of them. 

5 Unwinding Proof Technique 

We desire a technique for drawing conclusions about the global behavior (executions) of the system 
from local aspects (states, actions, and transitions) of the model. Faced with a similar situation, 
Goguen and Meseguer introduced unwinding relations to simplify proving that a system has non- 
interference |GM84j . We present a similar technique for proving that a system has differential 
noninterference. In particular we state what it means for a relation family to be an unwinding 
family and prove Theorem [2j which roughly states that the existence of an unwinding family for 
a given automaton implies that it satisfies differential noninterference. Our unwinding notion is 
probabilistic and approximate, which is in keeping with the notion of differential privacy. The 
novelty lies in the way we keep track of the privacy leakage bound, which evolves as the system 
evolves where the evolution is constrained by the differential privacy definition. 

5.1 Definition and Soundness 

Formulating a notion of unwinding relation that is sound for showing differential noninterference 
is more complicated than existing notions for showing noninterference because we must deal with 

We may also mechanically prove that these subroutine automata have differential privacy using other formal 
methods such a type system [RP10] , 
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probabilities and we must keep track of the privacy leakage bound e. To deal with probabilities 
and approximation, we adapt the notion of approximate lifting from previous work on approximate 
probabilistic simulation relations in the context of cryptographic protocols [ST07j . However, such 
work does not deal with tracking a leakage bound (see Section [7] for additional details). Thus, we 
introduce a family of unwinding relations indexed by various amounts of privacy leakage. Each 
unwinding relation in the family is a relation on the state space of the automaton. The unwinding 
relation indexed by the leakage amount e relates states that exhibit approximately the same trace 
distributions in the sense of e-differential noninterference. 

To deal with probabilities in a concise and modular way, we first define an approximate lifting 
operation that takes a relation over sets and produces a relation over distributions on those sets. 
The degree of approximation is governed by a parameter S. 

Definition 4 (5- Approximate Lifting). Let R be a relation between a set X and a set Y . The 
5-approximate lifting of R denoted by C(R,5) is the relation between Disc(A) and Disc(Y) such 
that for all v\ in Disc(X) and v 2 in Disc(Y), v\ £(R, 5) u 2 if and only if there exists a bisection 
(3 : Supp(^i) — > Supp(z/2) such that for all x in Supp(z^i), x R f3{x) and \ \n.v\{x) — In z^2(/3(x))| < 5. 

The requirement for /3 to be from the support set of v\ to the support set of ensures that 
if a state is assigned a non-zero probability in v\ then it is not possible for a related state to be 
assigned a zero probability in u 2 and vice versa — there is one to one correspondence between the 
states with non-zero and identical probabilities in the two distributions. The form of 5 involves 
natural logarithms because the privacy leakage bound in the differential privacy definition appears 
in the exponent. 

Next we define our unwinding technique, which is illustrated in Figure El Intuitively, since we 
want the behavior of the automaton to change only by a factor of e on receiving a single data point, 
we want the transitions under a data point from a state s to lead to states s' that are only a factor 
of e different from s. Covering (Definition [6]) formalizes this by requiring that state s is related to 
each such state s' by a relation lZ e that is part of an e-unwinding family (Definitional). 

In more detail, an e-unwinding family starts with a privacy leakage budget of e, which decreases 
over time to a current balance of e' . Related states s\ and S2 are required to only make transitions 
under the same actions. The distributions u\ and V2 that result from these transitions followed by 
any number of transitions under hidden outputs may differ only by a factor of 5. This difference 
is subtracted from the current balance e' to get a new current balance. Once the balance reaches 
zero, the resulting distributions must be equivalent. As the balance started at e, only a total of e 
privacy can be leaked, a point proved in Lemma [TJ 

Definition 5 (e-Unwinding Family). For a non-negative real number e, a family indexed by the set 
[0, e] of relations 1Z' over the H -disabled states of a puts L is an e-unwinding family for L if for 
all e' in [0, e], for all x± and X2 in S± such that x\ lZ e X2, for all a in I U R, there exists v\ such 
that x\ =1- v\ iff there exists V2 such that X2=^V2, and when they do exist, there exists a real number 
5 in [0, e'] such that u\ C(TZ e ~ s , 5) V2- 

Lemma 1. For all e-unwinding families 7Z' , all e' in [0, e], all x\ and X2 in S± such that x\ 1Z € ' X2, 
all i in I* , and all e in E* , both 

Pr[ [l(L,xMi)\ E □ e] < exp(e')Pr[ [1(L,x 2 )}(T)\e □ e] and 
Pr[ H(L,xM?)\e □ e] < exp(e')Pr[ [l(L,x 1 )j(i)\ E □ e\. 
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Figure 2: Unwinding Family and Covering: The left side shows the requirements for a covering. 
The right side shows the requirements placed on an unwinding family. The solid arrows denote 
the extended transition relation =>■ and clouds depict probability distributions such as v where 
s' e Supp(V). 



The above lemma shows that two states related by an e-unwinding family, given the same input 
sequence, produce distributions that only deviate by a factor e. Thus, to maintain e-differential 
noninterference, we desire that a state s should upon receiving a single data point d transition to 
a state s' that can be put into an e-unwinding family with s. We formalize this intuition with the 
next definition and confirm it with the following theorem. 

Definition 6 (Covers). We say that an e-unwinding family TZ' for a plts L covers a state s and 
data point d of L if 8=3* v implies that v(A-) = and for all s' 6 Supp(z^), s 1Z e s' . 

Theorem 2. For an automaton M = (L,sq), if for all H -disabled states s reachable from sq and 
all data points d, there exists a e-unwinding family that covers s and d, then \M\ has e-differential 
noninterference. 

Appendix [E] holds the proofs of Lemma Q] and Theorem [2 We prove Lemma Q] by induction 
over the structure of a. The interesting cases arise when a is of the form i:a' for i S / or o:a! 
for o £ O, which require similar reasoning. Suppose that a = i:af and s\ => v\ for some i G I. 
By the unwinding relation, we know that there exists a transition S2 z^2 such that v\ and V2 are 
in keeping with the privacy leakage bound imposed by the unwinding relation. Then for states 
s\ G Supp(z^i), and s' 2 6 Supp^), we apply the inductive hypothesis for a' to obtain the result. 

To prove Theorem [2j we use Proposition [12] and show for all i±, 12, and e where A(ii,«2) = 1 
that Pr[|_[(L, s)] (*i)J < exp(e)Pr[|_[(L,s)]](z2)j£;3e|- We use proof by induction over i\, 12, 
and e. When we reach the point where i\ and 12 differ by a data point d, we apply Lemma Q] 
knowing that an e-unwinding family exists for the current state s and d. 

5.2 Example: Applying the Proof Technique 

We now return to the parametric automaton model M ex i (K) of Section 13.31 We show that for any 
K, every state s and data point d of L ex i(K) can be covered by a (2t * e)-unwinding family 1Z' d in 
the sense of Definition [H Differential noninterference will follow from Theorem [2j 
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For the (2t * e)-unwinding family TZ' d , we construct for each j in [0, t] the unwinding relation 
TZ 2 JV '■ To construct these unwinding relations, we first introduce some notation. 

For a state s = {pc,B,n,c,y,r,k) and d £ D, add(s, d, d) adds d to the slot d of the state s. 
Formally, 

add(s, c , d) = (pc, B', ft, c, y, r, k) 

where B' = B and n' = n when n c = v and, otherwise, B' c = B C ^S {d}, n' c = n c + 1, and for all 
d 7^ c, f?^ = S c ' and n^,, = n c /. 

The function swap replaces one data point with another. Formally, 

swap(s, c , c£, d') = (pc, B 1 , n, c, y, r, /c) 

where = B d - {d f } l±) {d} and B' pll = B c „ for all c" / c'. 

For j such that < j < t, let 5^ to be the set of all states si such that si is reachable from s 
using t — j queries and any number of data points. Intuitively, this means that from s\ one can pose 
j more queries until the privacy budget runs out on the data point that is input into the system in 
state s. We define the relations as follows: 

• For j > 0, let TZ 2 /^ 6 to be such that for all si G S{, si TZ 2 J^ e add(si,c, d) and for all d', 

s\ Tl 2 /^ 6 swap(si, c, d, d') where s = (pc,B,n,c,y,r,k). That is, Tl 2 /^ 6 relates a state to the 
states it could have become had it received d as input when the cur Slot was c, the value 
curSlot had in state s. 

• For j = 0, TZ 2 /^ 6 is as above for states with a PC of 16 and is equality for those with a PC of 
08. 

Lemma 2. For all sets K of functions such that each function in K has e- differential privacy, for 
all states s and for all data points d, TZ' d is a (2t*e) -unwinding family for the automaton M ex i(K). 

Appendix IF] holds the proof. The proof uses a case analysis over the different types of actions 
a that might be received by two related states. The most interesting case is when a is a query and 
j = 1. In this case, s\ 7Z € S d S2 implies that s\ is in with s± and S2 reached in t — 1 queries. For 
a2t*e privacy leakage bound, this corresponds to the last time d may be used in answering a query. 
This requirement is met since for s\ and S2 to be reached with t — 1 queries, by the construction of 
A/exi {K)> curSlot in both states must be t — 1 slots away from the slot that holds d. Thus, after 
answering the next query the slot curSlot, whose value is always mod t, will point to the slot that 
holds d and that slot will be rewritten removing d. 

Since 7Z 2 ^ d e covers s and d for all states s and data points d of the automaton M ex i(K), Lemma[2] 
and Theorem [2] implies that the automaton has {2t * e)-differential noninterference. 

Theorem 3. For all set of functions K such that each function in K has e- differential privacy, 
M ex i(K) has (2t * e)- differential noninterference. 

As count and SUM are e-differentially private functions, this implies that M ex i ({count, sum}) 
has (2t * e)-differential noninterference. Furthermore, as explained in Section 14.31 subroutine com- 
position shows that M eX 2, a system with COUNT and SUM implemented as subroutines instead of 
atomic transitions, has (2t * e)-differential noninterference. Thus, we have proved that our exam- 
ple has (2t * e)-differential noninterference. In the next section we turn to mechanically verifying 
differential noninterference. 
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isUnwindFam((S', I, O, T),xel,5, t) 

convert all hidden actions of {{S,D,Q,R,T),so) to be the same one 
if(|rel| + 

return false 
for all i in [0, t], 

for all {xi,X2) G rel[z], 
for all a € / U O, 

if (T[zi][a] = nil xor r[xs][a] = nil), 

then return false 
if (T[si][o] / nil and T[x 2 ][a\ / nil), 
v\ = closure((5', I, 0,T),x\,a) 
i>2 = closur e((S, 1,0, T),X2, a) 
if(not isInLif tedRelation(5j_, rel[i], 0, v\, v 2 )) 
if(i = 0), 

return false 

if(not isInLif tedRelation(S'j_, rel[i — 1], 5, u\, U2)) 
return false 

return true 

Figure 3: Algorithm for checking relation families. 

6 Mechanizing Verification of Unwinding 

We provide an algorithm that soundly checks if a given family of relations is an unwinding family 
for a given automaton. While our algorithm does not generate the unwinding family, it automates 
the process of showing that a candidate family satisfies all the conditions for being an unwinding 
family (Definition [5]). By repeatedly applying our algorithm to a collection of relation families, 
we can algorithmically check that the covering condition of Theorem [2] holds and that automaton 
has differential noninterference. The process of verifying an unwinding relation family manually 
is typically tedious and sometimes error-prone. The existence of a mechanized verifier hence adds 
practical value to the proof technique presented in the previous section and justifies its use in favor 
of ad hoc proof methods. 

6.1 Algorithm 

Our algorithm isUnwindFam takes as input a labeled transition system of finite size, an array of 
relations over the system's states, a value 5, and a natural number t. The array rel may only 
represent relation families 1Z' over the interval [0, t * S) of a restricted form. 1Z' must be such that 
j^jS _ -j^kS £ Qr gji j anc j s UC h that [j\ = [k\ . That is, it must be possible to break the index set 
of 1Z' into t intervals of size 5 such that the relations in that interval are the same and one point 
corresponding to 1Z uS . 

The algorithm is shown in Figure O It represents the transition relation — > as an array T with 
rows and \A\ columns where T[_L][a] = nil for all a. The array either stores a distribution 
over next states or nil to indicate that the state cannot transition under that action. 

The first step of the algorithm converts all the hidden actions to be the same one since closure 
presumes just one hidden action. The function closure, defined in Section I4.lt computes the 
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isAllCovered(((S, I, O, T), a ), Rels, 5, t) 

reachableStates := computeReachableStates((S', D, Q, R, T),sq) 
for all s in reachableStates, 
for all d G D, 

if(T[ S p]/nil), 

v = closure((5, I, O, T), s, d) 
if(K-L) / or |Rels[s][d]| / t + 1), 

return false 
for all s' G 5, 

if(f(a') > and (s,s'} <£ Rels[s][d}), 
return false 
if(not isUnwindFam((5, /, 0, T), Rels[s][d], 5) 
return false 

return true 

Figure 4: Algorithm for checking for differential privacy. 

distribution over states that results from the system exhibiting the observable behavior a from a 
state X{ and computing until reaching an ^-disabled state. 

The distributions resulting from closure are compared with the provided family rel using the 
function isInLif tedRelation to determine whether they obey the requirements of a unwinding 
family. isInLif tedRelation(R, 5, v\, v-i) checks if the two distributions v\ and vi are related by 
the 5-approximate lifting of R. This function operates in OdS 1 ) 2 ' 5 ) time by reducing the problem 
to the decision problem of if a perfect matching exists for a bipartite graph, which can be solved in 
OdS*! 2,5 ) using the Hopcroft-Karp algorithm |HK73j . The reduction constructs a bipartite graph 
such that each vertex in the left part of the graph corresponds to a state in the support of u%, and 
each in the right part to a state in the support of U2 ■ Edges connect those states x\ in the left 
part to those X2 in the right part such that x\ R x<i and | lnz^i(xi) — lni^^)! < 5. A matching 
of graph that includes every vertex (i.e., a perfect matching) exists iff there is a bijection showing 
that v\ C(R,5) vi- Appendix iGl formally presents the algorithm and proves this result. 

The following lemmas state, respectively, the soundness and the runtime complexity of the 
algorithm. Appendix |H] contains the proofs for this section. 

Lemma 3 (Soundness). If the algorithm isUnwindFam(L, rel, 5, t) returns true, then z el corre- 
sponds to relation family that is (t * 5)-unwinding family for L. 

Lemma 4 (Runtime Complexity). The algorithm isUnwindFam runs in 0(t * \A\ * \S\ 4 ) time. 

We use isUnwindFam to construct an algorithm isAHCovered that checks a collection of relation 
families to conclude if they prove that an automaton has differential privacy (using Theorem [2]). 
In particular, the algorithm takes as input an automaton, an array Rels of relation families, 5, and 
the natural number t. For all states s that are reachable from the start state of the automaton 
and data points d, the algorithm uses isUnwindFam to check whether Rels[s][d] corresponds to a 
(t * <5)-unwinding family that covers s and d. The algorithm is shown in Figure [H 

The following theorems state the soundness and the runtime complexity of the procedure for 
checking whether all reachable states are covered by a given collection of relation families. 
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Theorem 4 (Soundness). 7/isAllCovered(il/, Rels, 5, t) returns true, then M has (t*5)- differential 
noninterference. 

Theorem 5 (Runtime Complexity). The algorithm isAHCovered runs in 0(t * \D\ * \A\ * \S\ 5 ) 
time. 

While sound, the algorithm is not complete even for this restricted class of unwinding relations 
it accepts as input. The algorithm (soundly) rejects any family if it has a relation that relates 
two states that transition to distributions over next states that differ by more than 5. That is, it 
requires that the automaton never leaks more than a 5 worth of private information in a single 
step. Furthermore, it pessimistically presumes that every leakage of private information is a whole 
5s worth. 

Nevertheless, we believe the algorithm is still of interest. In the next section, we show that 
it is powerful enough to prove that our example system, which is similar to pinq, has differential 
noninterference. While this system only has two very simple sanitization functions, COUNT and SUM, 
our algorithm will work for more complex sanitization functions provided they can be computed 
with a finite number of states. 

6.2 Example: Using the Algorithm 

To use our algorithm, we must first model the above program as an automaton M eX 2 with the 
subroutines COUNT and SUM in-lined as explained in Section 14.31 Then, we must construct Rels, 
which stores all the needed (2t * e)-unwinding families in the correct format. Such families exist 
since whenever M ex 2 leaks privacy, it leaks no more than 2 * e in a single step, and, thus, we can 
use 2 * e for 5. These families are instances of the parametric families shown in Section 15.21 The 
reader can confirm that these families may be expressed in the needed format for Rels. 

Indeed, as the body of the sanitization functions consists entirely of ff-enabled states, only the 
distributions over return values matter to our algorithm in that they influence the computation 
of closure and nothing more. Thus, the general families further shows that our algorithm can 
verify any modification of M eX 2 that substitutes a different set of e-differentially private functions 
for {count, sum} provided that those functions can be implemented using a bounded number of 
states as we would expect from the discussion of composition in Section 14.31 

7 Related Work 

Formal Verification of Differential Privacy. The most closely related work to ours is a 
programming language with a linear type system for proving that well-typed programs in the 
language have differential privacy |RP10| . Later work applies their type system to detecting network 
attacks in a private manner [RAW + IP] . The usual trade-offs between a program analysis technique 
designed to work over standard programming languages and a custom type system for a specialized 
language apply: the type system makes explicit in the source code why the program has differential 
privacy and type checking scales well, but the programmer must use a special-purpose programming 
language and annotate the code as the type system requires. Additionally, their programming 
language lacks I/O commands for creating interactive systems whereas our proof technique is for 
automata modeling interactive systems. 
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Other Differential Privacy Definitions. The definition of differential privacy may be seen as 
largely a simplification of the previously defined notion of e-indistinguishability [DMNS06], which 
explicitly models interaction between a private system and the data examiner as in our definition 
of differential noninterference. Our definition, however, is cast in the framework of probabilistic 
automata rather than Turing machines. This supports having structured models that are capable 
of highlighting issues arising from the bounded memory of actual computers. Furthermore, we deal 
with non-termination using prefixes allowing us to leverage previous work on formal methods for 
automata (e.g., |LSV07| ). 

Differential privacy is a very active research field giving rise to new definitions and techniques 
at a fast pace [DwolOl IDNPRIO] , For example, pan-privacy is a notion of differential privacy that 
gives differential privacy against adversaries that can observe the internal state of a system, in ad- 
dition to outputs [MPRV09] . Computational differential privacy gives certain differential privacy 
guarantees against computationally bounded adversaries. Our definition of differential noninter- 
ference and the formal proof technique was developed from the definition of Dwork |Dwo06j . We 
think that our choice of probabilistic automata as a model would prove useful in extending the work 
of this paper to these new definitions as well. For example, algorithms such as stream-processing 
algorithms that have been subject to research from pan-privacy point of view can be naturally mod- 
eled using probabilistic automata. Similarly, probabilistic automata-based models have successfully 
been used in the formal analysis of cryptographic protocols against computationally bounded ad- 
versaries [ST071 IBPW071 ICCK+08) . 

Information-Flow Properties. Differential noninterference has some similarities with informa- 
tion flow properties such as noninterference |GM82j . The literature contains several works on the 
use of transition systems, observational equivalences, and various notions of bisimulation relations 
to define information flow properties. To name a few, Focardi and Gorrieri have developed a clas- 
sification of noninterference-like properties in the unifying framework of a process algebra in a 
non-probabilistic setting [FGOlj . Sabelfeld and Sands [SSQOj . and Smith |Smi03j have used prob- 
abilistic bisimulation in defining probabilistic noninterference for multi-threaded programs, which 
they enforce using type systems. Probabilistic noninterference is regarded by many to be too strong 
in practice since it requires the probabilities of traces of the system observable by low-level users 
to be identical for any pair of high-level inputs (data points in our setting) |Gra91l IGra92j . As 
noninterference is often too strong of a requirement, weaker probabilistic versions have been pro- 
posed that allow for some information leakage [PHW04] IBP02j . Di Pierro, Hankin, and Wiklicky 
introduced approximate noninterference [PHW04] . and Backes and Pfitzmann introduced compu- 
tational probabilistic noninterference |BP02| . both of which allow for some information leakage. 
However, unlike differential noninterference, they do not allow the system behavior to diverge as 
the difference between the high-level inputs (data points) increases. This divergence, which is al- 
lowed by our differential noninterference definition (Proposition [13] in Appendix [C]) , is needed to 
release meaningful statistics and gain utility from the data set as discussed in detail in Section [TJ 
Quantitative information flow analysis attempts to determine how much information a pro- 
gram provides an adversary about a sensitive input or class of inputs. Clark, Hunt, and Malacaria 
present a formal model of programs for quantifying information flows and a static analysis that 
provides lower and upper bounds on the amount of information that flows [CHM07]. They measure 
information flow as the mutual information between the high-level inputs and low-level outputs 
given that the adversary has control over the low- level inputs. Malacaria extends this work to 
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handle loops |Mal07] , and Chen and Malacaria to multi-threaded programs [CM07] . McCamant 
and Ernst |ME07] . and Newsome and Song |NS08j provide dynamic analyses for quantitative in- 
formation flow using the mutual information formalization. There is also recent work on efficient 
computation of information leakage in the information theoretic-sense using a probabilistic automa- 
ton model [APvRSiO] , All of the above approaches assume that the adversary's beliefs are aligned 
with the actual distribution producing the sensitive input (s) and that adversary has no additional 
background knowledge. Clarkson, Myers, and Schneider instead propose a formulation using the 
beliefs of the adversary |CMS05j . However, such a formulation may be difficult to apply in practice 
because the surveyor may not know the adversary's beliefs. An advantage of differential privacy is 
that no assumptions are needed about the adversary's auxiliary information, computational power, 
or beliefs. 



Proof Techniques for Transition Systems. Simulation and bisimulation provide a system- 
atic proof technique for showing implementation and equivalence relationships between two au- 
tomata [Mil89l lLV95| ISL95] and are related to unwinding (see e.g., |BFPR03| ). Most similar to 
our unwinding technique, Segala and Turrini have studied approximate simulation relations in the 
context of cryptographic protocols |ST07| . Their work differs from ours by using asymptotic ap- 
proximations and only executions of polynomial length in terms of a security parameter. Their work 
allows certain transitions of the protocol to not have a matching transition in the specification. This 
models the capability of the adversary to compromise correctness. A protocol is deemed correct 
if the leakage accumulated at the end of a polynomial length execution is exponentially small in 
some security parameter. Our unwinding technique, on the other hand, requires that there always 
be an approximately matching transition, uses an exact error bound, and considers executions of 
any length. However, the probabilities of those transitions are only within some exponential mul- 
tiplicative factor of one another. Thus, neither approach subsumes the other. Furthermore, our 
relations are over states whereas theirs is over prefixes of executions. 

Much work has been done on decision algorithms for probabilistic simulation and bisimula- 
tion [BHK041 IBEMC001 IPLS001 ICS02] . Particularly relevant are the works of Baier and Her- 
mans |BHK04j . and Cattani and Segala [CS0 2] on decision algorithms for weak bisimulations. 
Since our unwinding relations keep track of an error bound in the form of indices in a relation 
family, the methods of these papers to generate relations do not readily apply to our setting. We 
limit ourselves to checking if a given relation family is an unwinding family rather than generating 
one. Extending these prior works to our setting remains as future work. 

Finding refinement metho ds that preserve information flow properties has been investigated by 
several authors [ManOH \J0l\ IHPS014 IAvZ06j . In most of those works refinement is used in the 
sense of reducing various flavors of nondeterminism in an abstract system. For example, Mantel 
focuses on a range of information flow properties and unwinding conditions as local conditions that 
imply these properties |Man01j . He then presents some operators that refine a given transition 
system such that these conditions are preserved in the system refined by the given operators. We 
have a more restricted goal in this paper, namely, to pin down the conditions under which an 
abstract internal transition can be replaced by a sequence of internal transitions in a way that 
will preserve differential noninterference. This is sufficient for our purposes because such transition 
replacements are the sources of different abstraction levels that typically arise in the analysis of 
systems we consider in this paper. 
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8 Future Work 



The results of this paper represent progress towards developing a basis for the formal verification of 
differential privacy for systems, but leave open several interesting directions that we plan to explore 
in future work. We hope to create a decision procedure for our proof technique by extending prior 
work on decision procedures for probabilistic bisimulations [BHK04. BEMCOO l IPLSOOl ICS02| to 
make them produce a family of relations rather than a single one. We also plan to extend the 
theory to model and reason about higher level systems, such as computer systems of hospitals 
and other distributed systems [RRS + 10] that allow interactions of the system with data providers 
and with data analysts, while protecting the privacy of the data stored and manipulated by the 
system. For example, airavat allows computations over data distributed in a cloud, and combines 
mandatory access control with differential privacy where differential privacy is used to facilitate 
declassification governed by the privacy error bound set by a data provider. Our techniques can 
currently apply to the verification of differential privacy property of the AIRAVAT system using a 
whole-system model. We are interested in exploring the computational model of AIRAVAT further 
to understand the interplay between the fine-grained access control mechanisms and the differential 
privacy mechanisms in stating the end-to-end information-flow guarantee of AIRAVAT. Moreover, we 
wish to extend compositionality aspects of our framework so that we can decompose the reasoning 
about such properties, and exploit our proof technique for differential noninterference for parts of 
the proof. Finally, while the current paper uses manually constructed automata models of systems, 
we plan to develop techniques to extract such models from source code of software systems such as 
PINQ [McS09| and airavat [BRS+IO] . 
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A The Truncated Geometric Mechanism 
A.l The Mechanism 

The Truncated Geometric Mechanism of Ghosh et al. [GRS09J is an adaptation of the Laplace 
mechanism made to produce outputs over only a bounded range of discrete values. The Laplace 
mechanism works by computing the exact result of some statistic / and then adding noise drawn 
from a Laplace distribution. The amount of noise depends upon both the privacy parameter e and 
the sensitivity of /. The sensitivity of / is the amount the value that / computes can change by 
adding or removing a single data point from the data set. Formally, the sensitivity of /, denoted 
5(f), is maximum value that \f(B\) — /(i?2)| can take on where B\ and E>2 ranges over all pairs of 
data sets differing by one data point. Using rej^ to denote the Laplace mechanism applied to the 
statistic /, we have that Kj^(B) = f(B) + Lap(5(f)/e) where Lap(6) is a random variable producing 
noise according to the Laplace distribution centered at zero with variance 2b 2 . 

To make the Laplace distribution discrete, start by noting that informally the Laplace distribu- 
tion is two exponential distributions back to back. That is, Pr[Lap(6)=x] = Pr[Exponential(l/6)=|a;|] 
where Exponential(A) is the exponential distribution with the p.d.f. of Aexp(— Xx) at x for x < and 
otherwise. Since the discrete version of the exponential distribution is a geometric distribution, 
one can use two geometric distributions back to back to create a "discrete" Laplace distribution. 
Formally, Pr[Exponential(A)=x] = Pr[Geo(exp(— A))=[xJ] where Pr[Geo(p)=/c] = p k (l — p) (i.e., p 
is the "failure probability"). Using DL to denote this distribution, we have that Pr[DL(p)=n] = 

p i+p- 

Next, one must bound the mechanism to produce only results between the minimal and max- 
imum numbers that the computer can represent. For simplicity we assume that the minimum 
is — m where m is the maximum. Thus, we need that the result of adding noise f(B) + A is 
such that —m< f(B) + N < m where ./V is random variable generating noise. This implies that 
— m — f(B) < N < m — f(B) requiring that N depends upon both m and f(B) in addition to e 
and <*(/). ' 

At this point, it may be tempting to simply take the discrete Laplace distribution DL and 
condition on the noise being between —m — f(B) and m — f(B). This will produce a bounded 
distribution such that the probability of producing two adjacent outputs are within a multiplicative 
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factor of one another. However, since the condition involves the value of f(B), the distributions 
resulting from two adjacent data sets may differ. In general, they need not be within a multiplicative 
factor of one another. 

Fixing this problem requires adding extra weight to the probability of producing the extreme 
results — m and m for f(B) + N. Intuitively, this extra weight account for the tails being cut off. 
Formally, it comes from a system of equations constraining the relationship between each pair of 
distributions N(m, f(B\), exp(— e/6(f))) and N(m, f(B 2 ), exp(— e/6(f))) where B\ and B2 differ 
by one data point. Formally, 

{p' n ' * iq^ I* + n \ = 771 
^' n ' * TT§ -m<t + n<m 
otherwise 

N produces noise for differentially private mechanism for the statistic /: 

k u (B) = f(B) + N(m, f(B), exp(-e/<y(/))) 

Proposition 1 (Differential Privacy). For all integers m > 0, for all functions f from data sets to 
{—m, ... , m}, the function Kf te has e- differential privacy. 

Proof. By a lemma similar to Proposition 1121 since k/ j(E is discrete, it gives e-differential privacy iff 
for all data sets B\ and B2 differing on at most one element, and for all r € range(ftf je ), 

Vt[k u {B x ) =r}< exp(e) * Pr[« /)e (B 3 ) = r] 

Note 

Pr[« />6 (fl) = r] 

= Pr[/(J3) + N(m, f(B), exp(-e/<5(/))) = r] 
= Pr[iV(m, /(B), exp(-e/<J(/))) = r - f(B)] 

'exp(- e /5(/))l-/( B )l * m^T^ \f(B) + (r- f(B))\ = m 
exp(- e /J(/))l-/( B )l * ^g§E%jj -m < /(B) + (r - /(B)) < m 
otherwise 



r = m 



exp(-|r-/(B)|6M/)), 1+cxp( i 

exp(-|r - f(B)\e/8(f)) * ^(-f/lf/)) < r < m 

otherwise 

Thus, if r > m or r < — m, Pr[K/ |£ (-Bx) = r] = < = exp(— e) * Vx[Kf^(B 2 ) = r]. Otherwise, 
since the normalization factor, which depends on whether |r| = m or not, is the same on each side 
of the inequality Pv[Kf^(Bi) = r] < exp(e) * Pr[Kf^(B 2 ) = r], the inequality holds iff 

e-\r - f(B 1 )\e/6(f) < exp(e) exp(-|r - f(B 2 )\e/5(f)) 

Since B\ and B2 only differ by at most one data point, we know that \f(Bi) — f(B 2 )\ < 6(f). 
Case: f(B 2 ) < f(B l ). In this case, f(B x ) - f(B 2 ) < 6(f). Let f(B x ) - f(B 2 ) = d so that 
exp(-|r - f(B 1 )\e/6(f)) = exp(-|r - (f(B 2 ) + d)\e/6(f)). 



29 



• Subcase: \r - f{B 2 )\ < \r - (f(B 2 ) + d)\. In this case, exp(-|r - (f(B 2 ) + d)\e/5(f)) < 
exp(-|r- f(B 2 )\e/5(f)). Thus, 

exp(-|r - /(SOle/^/)) < exp(e) exp(-|r - f(B 2 )\e/6(f)) 

since e < 0. 

• Subcase: \r - f(B 2 )\ > \r - (/(B 2 ) + d)\. Let & = \r - f(B 2 )\ - \r - (/(B 2 ) + d)\. Since 
ff < d, 

exp(-|r - /(Bi)|e/*(/)) = exp(-(|r - (/(B 2 )| - &)e/8(f)) 

= eM(d'-\r-f(B 2 )\)e/S(f)) 
<eM(d-\r-f(B 2 )\)e/6(f)) 
<eMW)-\r-f(B 2 )\)e/5(f)) 
= eMWWf))-(\r-f(B 2 )\e/S(f))) 
= ex P (e-(|r-/(B 2 )|e/<5(/))) 
= exp(e)exp(-|r-/(B 2 )|e/<5(/)) 

Case: /(Bi) < f(B 2 ). In this case, -(/(Bi) - /(B 2 )) = /(B 2 ) - /(Bi) < <J(/) Let /(B 2 ) - 
/(Bi) = a so that 

exp(-|r - f(B 2 )\e/S(f)) = exp(-|r - (/(B^ + 0)|e/*(/)) 

• Subcase: |r - /(Bi)| < |r - (/(Bi) + Let 5' = |r - (/(Bi) + 0)| - \r - f{B{)\. Since 

a' < a, 

exp(-|r - /(Bi)|e/5(/)) = exp((-|r - /(Bi)|e - 5(f)e + S(f)e)/8(f)) 

= exp(((-|r - /(BO) - <5(/))e + eS(f))/S(f)) 
= exp((-|r - /(BOI - 8(f))e/S(f) + eS(f)/6(f)) 
= exp((-\r-f(B 1 )\-5(f))e/5(f) + e) 

< exp(e) exp((-|r - /(Bi)| - S(f))e/5(f)) 

< exp(e) exp((-|r - f(B 1 )\ - d')e/5(f)) 
= exp(e)exp(-(|r - f( Bl )\+d')e/5(f)) 
= exp(e) exp(-|r - (f(B 1 ) + 3)|e/J(/)) 
= exp(e)exp(-|r-/(B 2 )|e/<5(/)) 

• Subcase: \r—f{B\)\ > \r—(f(Bi)+d)\. In this case, we have that exp(— |r — (f(Bi))\e/5(f)) < 
eM-\r-f(B 1 ) + d\e/6(f)). Thus, 

exp(-|r - f(B 1 )\e/5(f)) < exp(e) exp(-|r - f(B 2 )\e/5(f)) 

since e < 0. 

□ 

The probability of Kf e (B) being b or more away from f(B) decreases exponentially in b. 
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Proposition 2 (Utility). Pr[|« /i6 (J3) - f(B)\ >b}<^. 
Proof. 

Pr[\K U (B)-f(B)\>b] 

= l-P T [-b<K U (B)-f(B)<b] 

= 1 - Pr[-6 + 1 < Kf t6 (B) - f(B) <b-l] 

= 1 - Pr[-6 + 1 < f(B) + N(m, f(B),exp(-e/8(f))) - f(B) < b - 1] 
= 1 - Pr[-6 + 1 < N{m, f(B), exp(-e/5(/))) < b - 1] 
6-1 

= 1- Y, Pr[JV(m,/(B),exp(-€/5(/))) = n] 

n=-b+l 

If b - 1 > m - /(B) and -6 + 1 < -m - i, then this is 1-1 = 0. If b - 1 < m - /(£) and 
— 6 + 1 > — m — f(B), then this is 

i- £ ex P (-^ (/ )r ;- exp ;- e /^; 

n=^fl l + exp(-e/5(/)) 

1 + exp(-eW)) - 2exp(-e/^(/)) b 
l + exp(-e/*(/)) 

2p 6 



1+P 

If 6 — 1 < m — f(B) and —6 + 1 < — m — t, then this is 

l + exp(- e /S(/))-exp(- e /5(/)) 6 



1 



l + exp(-e/5(/)) 

6 



1+P 

If 6 > m — /(-B) and —6 > — m — t, then this is 

"v 1 exp(-^ (/ )r ;- exp i- e ^; 

1 +P n ±X+i l + exp(-e/<5(/)) 

l + exp(-6/<5(/))-exp(-e/ ( 5(/)) 6 



1 



l + exp(- e /5(/)) 



p 6 



1+P 

completing the proof. □ 
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A. 2 An Implementation 

Below is an efficient algorithm for sampling from N(m, t,p) for m > 0, — m < t < m, and < p < 1: 

01 sample_N(m,t ,p) 

02 if (flip(p/(l+p))) 

03 if (flip(p~(m+t-l))) 

04 return (-m-t) ; 

05 else 

06 q := (p-l)/(p-(m+t)-p) ; 

07 for(n:=-l; n>-m-t+l; n— ) 

08 if(flip(q)) 

09 return (n) ; 

10 q := p*q/(l-q) ; 

11 return (-m-t+1) ; 

12 else 

13 if (flip (p~ (m-t))) 

14 return (m-t) ; 

15 else 

16 q := (p-l)*p~t/(p~m-p~t) ; 

17 for(n:=0; n<m-t-l; n++) 

18 if(flip(q)) 

19 return (n) ; 

20 q := p*q/(l-q) ; 

21 return (m-t- 1) ; 

Each flip command uses an independent Bernoulli distribution to select either true or false. 
flip(p) returns true with probability p. 

Proposition 3 (Correctness). sample_N samples from N(m,t,p). 

Proof. Let q n denote the value that variable q has the beginning of the nth iteration of the last 
for loop: go = pmzzEi and q n = p* q n -\/(l — q n -i) for < n < m — t — 1. We show by induction 
over n, that for n between and m — t — 2, 

\ _1 n_1 1 

P \ /i m-twlTT/i „ \-l n i— P 



1 ; j=o 



l + p 



For the base case with n = 0, 

(p - iy 



p m — p i 



\ — i i 
(i-p™ - ^" 1 v 



i+pj i+p 
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For the inductive case, assume this is true for n — 1. Then, 

q n =P* - q n -i) 



= 1- 




1 +p 



J'=0 

rn-1. 



If the last for loop executes, then with probability 11^=0 (^-~ 1j)ln it will stop at the nth iteration 
and return n for values of n between to m — t — 2 (inclusive). Using above equation, 




m— 1\— 1 



1 n-1 1 _ 

(l-p-r'na-^rVy-f 

,n 1 ~ P 
l+P 



Since the probability of the for loop executing is ^1 — j+^j (1 — p rn *), this implies that the 
probability of returning n such that < n < m — t — 2 is p n jq^ = 



For < n = m — t, the probability of returning m—t is (1 — j^)p m = p m = P 



P \ r . m ~ t — n m ~** 1 — nl m- *l* 1 

[+^JP —P *j+^-P *— } 
The probability of the for running until completion and returning m — t — 1 is equal to the 
probability that none of the other values of n is returned. That is, the probability flip(p/ (1+p)) 
returning false less the probability of some other number between and m — t being returned: 



Nearly the same reasoning shows that the negative values for noise also have the correct prob- 
abilities. □ 

Assuming that all the operations in sample_N including flip are constant time, sample_N runs 
in expected constant time. 

Proposition 4 (Runtime Complexity). sample_N runs in 0(1) expected time. 

Proof. The expected running time is Ylri=-m-t P r [-^(m, t,p)=n] * T n where T n is the running time 
of sample_N when it produces n. The running time is constant in the case where sample_N produces 
m — t or —m — t. The running time T n is 0(|n|) for n such that — m — t < n < m — t. Thus, 



33 



ignoring constants, the expected running time is 



m-t-l 



^2 p |ri| - — -n < - — - * 2 * ^max(| - m - t + l|,m - t - l)p n n (1) 



1 + p 1 + p 

n=-m-t+l y 1 n=0 

1 — p 



< ! - * 2 * V oop n n (2) 

1+p 

*2* (3) 



1+p 1 — p 
2p 



(4) 



1+P 

<2 (5) 

where line [3] follows from the expected value of the geometric distribution. (Recall that we are 
using p to denote the failure probability unlike most references, which use 1 — p for the failure 
probability.) Thus, it is expected to run in constant time. □ 

A. 3 Using the Mechanism for the Sanitization Functions COUNT and SUM 

We use the above privacy mechanism to implement sanitization functions similar to the ones that 
pinq provides. Due to space constraints, we focus on two representative ones: count and SUM. 
Since we use a bounded discrete privacy mechanism over integers, our implementations differ from 
the implementations found in pinq. We force data points to be integers between —100 and 100 
whereas pinq bounds the sensitivity of functions by mapping data points to doubles between — 1 
and 1. (Our range may be made larger without affecting our results.) We then use numbers outside 
this range to encode objects other than data points such as queries. 

Given these, we implement our PiNQ-like system as follows, datapoint (y) on line 09 of the 
code of Figured] would be implemented as a function with body return(-100 <= y && y <= 100). 
emptyArray (x) must be implemented to store a value outside of { — 100, . . . , 100} so that data points 
can be distinguished from empty spots. The program uses numbers larger than 100 to indicate 
queries: 101 denotes COUNT and 102 denotes SUM. Given 101 or 102, get_sanitization_f unct (y) 
returns a function that computes the count statistic or sum statistic, respectively, count is com- 
puted with 

01 count (dPts) 

02 count := 0; 

03 for(j:=0; j<t; 

04 for(k:=0; k<maxPts; k++) 

05 if (-100 <= dp[j] [k] <= 100) 

06 count ++ ; 

07 else 

08 break; 

09 s := t*maxPts/2; 

10 noise:=s+sample_N(s,count-s,exp(-e/l)) ; 

11 result : =count+noise ; 

12 return (result) ; 



34 



and SUM with 

01 sum(dPts) 

02 sum := 0; 

03 for(j:=0; j<t; j++) 

04 for(k:=0; k<maxPts; k++) 

05 if (-100 <= dp[j][k] <= 100) 

06 sum := sum+dp[j] [k] ; 

07 else 

08 break; 

09 noise := 

sample_N (t*maxPts*100 , sum, exp (-e/100) ) ; 

10 result := sum+noise; 

11 return (result) ; 

where sample_N is as defined above and e stores the value for the privacy bound e. We add and 
subtract t*maxPts in the calculation of the noise in count to shift the noise over to keep the value 
count positive. 



B Automaton Model 

B.l Probability of Action Sequences 

We use (L, s)(i)(a, s') to denote the probability of the automaton (starting in state s) producing 
the trace a and ending in the state s' after producing the last action of a given that the available 
inputs are i. (L, s)(i)(a, s') is defined as follows: 

(L, s)(i:i)(i:a, s') = fi(s")(L, s"}(i)(a, s') ifsA/i 
s"es 

(L,s)(i)(o:a,s') = £ fi(s")(L,s")(i)(a,s') if sA/i 

s"es 

(L,s)(i)([],s) = l 

{L, s)(i)(a, s') = otherwise 

where i £ I and o G O. The first line in the above definition, for example, considers the case 
where the state s transitions to a new state under the input i according to the distribution fj,. It 
states the probability of starting in the state s, consuming the input i, and then performing the 
actions a ending in state s' given that i remain available inputs. This probability is the sum of the 
probabilities of transitioning to a state s" and then performing the actions a from s" , ending in 
state s' given that i are available inputs. 

Proposition 5. For all automata (L, s), a in A* , s' in S, and i in I* , (L, s)(i)(a, s') is well defined 
and between and 1. 

Proof. Proof by induction over the structure of a. 

Case: a = []. (L, s)(i)(a, s') is 1 if s' = s, and (L, s)(i)(a, s') is for s' / s. 
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Case: a = i:a! . If there does not exist i such that i = v.i and sA/j, then (L, s)(i)(a, s') = 0. 
If there does exist such a i', then (L, s)(i)(a, s') = Yl s "eS / u ( s ")(^i s")(i')(a!). By the inductive 
hypothesis, (L, s")(i')(a' , s') is well defined and between and 1 for all s" . Since /i is a distribution 
over states and the events of being in a state are mutually exclusive, ^s"gs^( s ") = 1- Let 
•Smax = argmax s » gS (L,s")(i')(a / ,s / )- 

l(L,s)](?)(a,s') = E M ( S '0(L, S ')(?)(^^') 

s"GS 
s"GS 

= (L,S max )(i)(tf,s') 
< 1 

Case: a = o:a' '. If there does not exist /x such that s4/i, then (L, s)(i)(a, s') = 0. If there 
does, then (L, s)(i)(a, s') = XL"eS s ') an d we can use the inductive hypothesis 

as above. □ 

A helpful proposition about our model follows. 

Proposition 6. For all PLTS L, states s',s" G S, i' G I*, and /i G ii"*, 

Proof. Proof by induction over the structure of h. In the case where h = [], (L, s") ([])(h, s'") = 1 
when s 1 " = s" and otherwise. Thus, 

^(L, S '0([])([], S ' / 0*(^O(^(«'^') = l*(^« // )([]:?)(a, S ') 

s"'eS 

= (L,0(?)(^a') 

Case: /i = h:h' for some /i and h! . If s" A //, then 



s"'eS 



Y / (L,s'n])(h:h\s' , ')*(L,s''')(?)(a',s') 

E f E mV")<£, W) * (a oflVy) 

s"'eS Vs""G5 / 

E m'co E( l ' sW/ )([])(^ / ^ /,/ )*(a s //, )(?)(3 / ^ / ) 

s""G5 s'"G5 

E ^Xs'''')(L,s'''')0 y )(h':S',s') 



s""GS 



(L,s")(i/)(h:ti:a?,s') 
com the inductive hyp 

= Es>»es( L , s ")([])(h,s'") * (L,s'")(i')(a',s>) since (L, s")([])(h, s'") = for all a"' G S. □ 



where third line follows from the inductive hypothesis. If for no //, then s" 4//, (L,s"){i'){h:a',s') 
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B.2 Extended Transitions 

We define s=>v so that u{s') is the probability of reaching the //-disabled state s' from the state 
s where a is the action performed from state s: 

u(s') = ^2 li{s") ( L > s ")(U)ft,s') s' is //-disabled 

s"€S heH* 

and v(s') = otherwise where s4/i. Thus, the probability of reaching the //-disabled state s 1 
from s by performing the action a followed by a sequence of hidden actions h is calculated by 
considering each s" that is reachable by performing the single action a from s. For each such s" 
we multiply the probability of ending up in s" by performing an a from s with the the probability 
of reaching s' from s" by performing a sequence of hidden actions (the inner sum). The value v{s') 
is then calculated by adding the probabilities corresponding to each s" . Since all h in H* contain 
only actions from H, an execution with the action sequence h cannot leave an //-disabled state. 
Thus, v(s') is the probability of s' being the first //-disabled state reached. If there is no fi such 
that s A fj,, then there is no v such that s =4> v. 

For notational convenience we extend the transition relation — > to S± by having no transitions 
to nor from _L This implies that 

<£,!_)(?)([],!_) = 1 
(L,s}(i)(a,±) = l-Y(L,s)(i)(a, S ) 

(L,±)(T)(a,x) = if a ^ [] or x ^ ± 

Thus, Pr[ [_l(L, _L)](i)j£;3e] is 1 if e = [] and otherwise, which matches the intuition that a 
nonterminating program which never interacts with the data examiner will only have the empty 
trace as a prefix. 

Proposition 7. For all states s and actions a, s=>v implies that v is a distribution over S±. 

Proof. To prove that v is a distribution over Sj_, we must show that for all x G S±, < v{x) < 1 
and Ylxes ± u ( x ) = 1- We start by proving that X^eS' — 1 by introducing a function ij. 
Given the set 5' of //-disabled states, let n be defined as follows: 

77(77., s) = 1 s G S 1 

77(77,, s) = when n = and s ^ S 

77(77,5) = fih{s")r]{n — l,s") otherwise 

heH s"es 

where s -A /i^ and 77 is a natural number. 

Proof by induction over 77 shows that 77(77,5) = YlheH^ n Es'eS'^A s ')- where //- n = 

/J n U Z/^™" 1 for 77 < 1 and //^° = H° = {[]}. In the base case, n = 0, if s G S", then 77(77, s) = 
1 = E s ' 6 S'CM>([])([]y) since (L, «>([])([], a ) = 1, (L, s) ([})([], s>) = for s>, and 8 G S'. If 
s ^ 5', 77(77, s) = = E s / g g/(£, «)([])([], s') since (L, «)([])([], s') = for s ^ s' and s ^ S' whereas 
s' G S'. 
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In the inductive case, if s £ S', then (L, s)([])(h, s') = if h 7^ [] or s' 7^ s since s is ii-disabled. 
Thus, r/(n + l, S ) = 1 = Efej<«+i E s ' eS '(^ «>([])& « ince <£,*>([])([],«) = L If « £ ^ t^n 

r/(n + 1, a) = E E ^( s X n > s") (6) 
heHs"eS 

= E E^( s ") E E^xuxmo (?) 

heH S "es h<z H < nS '£S' 

= E E E E/^'O^MaiXMO ( 8 ) 

s'6S' h€H£ eH <ns"eS 

= E E E <A«>([])(fc/h«') (9) 

= E<E E CMUIX^.*'))) + «>([])([],*') (10) 

= E E ^MawM') (ii) 
= £fo «>([])& «0 a 2 ) 

^ g J^<n+l s'eS" 

Line [7] follows from the inductive hypothesis. Line [9] follows since s is JT-enabled. Line [10] follows 
from (L,s)([])([],s') = since s $ S'. 

Induction over n can also show that < 77(71, s) < 1 since /i/j is always a distribution. 

We use 77 to show the following: 

E = E E ^ s ") E (^"xnXMO 

s'eS' s'eS" s"es f ieH * 

= E^ s ") E E^'xnxM') 

,"65 Kef/* *'eS" 

= emo 1™ E E^'xnxMO 

= > hm 7/(71, s") 

s"es 

< E moi 

s"es 

< 1 

where s4/i. 

For all s' € S, if s' is //-enabled, v(s') = 0. Thus, E s gs u ( s ) = Es'gS" K s — 1- Furthermore, 
for all s G 5, < i/(s) and < Esgs z/ ( s ) si nce 110 operations that could introduce negative numbers 
is every used in computing v (s). Since z^(J-) = 1 — E s gs K s )> — u {^-) — 1 an d Exe5 x ^( x ) = 1' 
Since for all s, < and EsgS z/ ( ,s ) — 1' ^ rnust be the case that v(s) < 1. □ 

Given such an automaton M = (L, s), we define [M] to be a function from input sequences to 
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a distribution over trace prefixes (finite action sequences). 

Pr[[M](?)3a] = ^M(?)(a, S / ) 

s'es 

We write \_cl\e for restricting the action sequence a to some subset E of A. Formally, \\}\e = [], 
[a:a\E = a: [S\e if a G F, and |_a:aj£ = [S\e, otherwise. For infinite sequences a with only a finite 
number of elements from E, \a\ e is the finite sequence that results from [a'\ e where a' is the finite 
prefix of a holding all the elements from E. If a contains an infinite number of elements from E, 
then [a\ e is the infinite sequence whose jth entry is the jth element of E in a. 

Given an automaton M, Pr[UMj(i)j£ □ e] is the probability of the data examiner seeing 
e G E* as a prefix given that the available inputs are i. To calculate Pr[[|[Mj(i)J_E □ e], consider 
the set 7(e) of action sequences a such that \cl\e = e and ends with the last element of e. That is, 
7(6*) = {a € A* \ \cl\e = e A last(a) = last(e) } with the special case that 7([]) = {[]}. To calculate 
Pr[L[A^](i)jE □ e], we need not consider all o such that \cl\e = e. Rather, we may focus only on 
those in 7(e) since every a such that [3,\e = e will have a prefix in 7(e). Since it is impossible to 
see two different prefixes from 7(e) during the same execution (no element of 7(e) is the prefix of 
another), they are mutually exclusive. Thus, Pr[|_[M](?)J.E □ e] = Y2ae-y(e) ^MI^Ow 3 &\- 

B.3 Some Helpful Propositions 

We need some propositions about our model to prove the soundness of unwinding later in Ap- 
pendix [El 

Let H*:^) stand for { a G A* | 3h G #*, 3a" G 7(e"), a = /J:a" }. 

We use 7'(e) do denote those action sequences of 7(e) that do not start with a hidden output 
from H: j'(e) = { a G 7(e) | a = [] \J 3a £ A - H,3a" £ A* ,a = a:a" } where A - H is the set 
difference. 

Proposition 8. For aZZ e G F* ; if e / [], i/ien H*:^'(e) = 7(e). 

Proof. To show that H*:^'(e) C 7(e), note that for all a G H*:^'(e), there exists /i G i?* and 
a' G 7'(e) C 7(e) such that a = h:a'. Furthermore, [h:a'\E = [(i'\e = since H n F = 0. Since 
e / [], a / [] and last(/i:a') = last(a') = last(e). Thus, h:a! G 7(e). 

To show that 7(e") C H*:^'(e), for any a G 7(e y ), either a G i?* or there exists h G i/*, 
a G A — H, and a' G ^4* such that a = h:a:a'. The first case cannot arise since it would imply 
that e = [] since e = [S\e = []■ For the second case, since e = [h:a:a'\E = [a:a'\E and last(e) = 
\ast(h:a:a') = last(a:o*'). Thus, a:a' G 7(e). Thus, h:a:a' G H*:^'(g). □ 
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Proposition 9. 

s'es 

i/sA/i and i G / 

Y,Ks')Pr[l(L,s')jmS] 

s'eS 

if sA- and o € O 

1 

otherwise 



s"£S 

Y, E^CMC^") 

s"£Ss'£S 

j>( S 0E^ s >«(^") 

s'GS s'GS 

moo p^[(i, «>feff] 



^(L, S )(r)(o:a, S ") 
s"eS 

= E £mOO<£, o 

s"eSs'eS 
s'GS s'GS 

= ^ M ( S 0Pr[(L, S }(^a] 

s'GS 

For the third equation: Pr[ {(L, s) J (»)□[] ] = Es'gs( L > s )<aKIL s ') = 1 since ( L > s )(^*)([]> s ) = 1 
and (L, a)(t)([], a') = for all s' / s. 

For the forth equation: Pr[ [<L, s)](?ga] = J2 s 'es( L > s )(^)([]> s = since ( L > «>(*)([]. s ') = 
for all s' . 

To show that < Pr[ [(L, s)J(i)3a] < 1, we use proof by induction over the structure of a. 
Case: a = []. [(L, s)}(i)(a) is 1. 

Case: a = v.a! . If there does not exist i' such that i = and s-V fj,, then [(L, s)J(i)(a) = 0. 
If there does exist such a i', then |(L, s)}(i)(a) = Ss'gS ^( s ')[K-k) s ')KO(^')- By the inductive 
hypothesis, {(L, s')J(i')(af) is well defined and between and 1 for all s'. Since ji is a distribution 



Pv[l(L,s)j(i:i)Ui:a] 



Pr[[(L,a)I(i)3o:3] 



Pr[ [<£,*>](?)□[]] 
Pr[[(L,s)](<)33] 

and < Pr[[(L,«)](t)3ff] < 1. 
Proof. For the first equation: 

PT[l(L, S )}(i:T)Ui:a] 



For the second equation: 

Pr[[(L,,s)K03o:a] 
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over states and the events of being in a state are mutually exclusive, ^s'es M s ') = 1- s rnax = 
argmax s , e5 [(L, s')}(i')(a'). 

[(L,«)I(i)(a) = ^^')[(^s')K?)(«) < w)K?)(«') 

= [(^,w)](?)(a) 
< 1 

Case: a = o:a'. If there does not exist [i such that s4/i, then |(L, s)](i)(a) = 0. If there 
does, then \(L, s)J(J)(a) = Yls'eS M s ')[I(-k> S ')J(J)(3,') an d we can use the inductive hypothesis as 
above. □ 

Proposition 10. For all H- disabled states s, a in DUQU R, e in E* , andi in I* , if e ^ [], s4/i, 
and s => v, then 

E E E mo *<ao (*)&*') = £ "(»)pr[L[(£,i)ito 



Proof. 



E E E^(0*<£, o (13) 

aG 7 (e) s'eSs"eS 

= ££MO E (L,s")(i)(a,s') (14) 

s'GSs"GS aG7(e) 

= ££mo E MftM as) 

s'eSs"eS aeH*:-y>(e) 

= E E MO E E (L,s")(i)(h:a,s') (16) 

= E E mo E E E (^oaDCM'") * (^'"x^o (17) 

s'GS s"eS ftgif* a£7(e) s'"eS 

= E E mo E E E (is) 

s'eS s"£S h;z H * aG 7 (e) s"'eS' 



E 


( £mo E< L 'Oai)(^o 


) E E^O^M 


(19) 


i"'GS" 




> aS7(e) s'GS 




E 


MO 2 Pr[[(L, S "')] (?)□«] 




(20) 


i'"GS" 


aG7(e) 






E 


MO IMLKAOI 




(21) 


E 


i/(x)Pr[L[(L,x)K?)jE3e] 




(22) 



where S' is the subset of states S that are .ff-disabled. Line 1151 follows from Proposition [5J Line [TBI 
follows since there is a one-to-one correspondence between elements of H*:^'{e) and H* x j'(e) given 
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as a G H*: r y'(e) corresponding to (h,a) where h is the largest sequence of H* such that a = h:a! 
for some a'. Line 1171 follows from Proposition [U Line 1181 follows since a G 7'(e) starting with an 
action not in H implies that (L, s"')(i)(a, s') = for any state that is -ff-enabled. Line 1221 follows 
from Pr[ [l(L, _L)](i)j£ZJe:e] = since e:e/ [] and v{s'") = for any if-enabled state s'" . □ 

Informally speaking the following proposition shows how we can account for transitions on 
hidden actions in calculating the probability of observing a particular behavior from a given state. 
The first part of the proposition states that the probability of observing the sequence e starting 
from the state s given the input sequence d:i' can be calculated by considering those states that 
are reachable from s by performing the action d followed by a sequence of hidden actions. For 
each such reachable state we take the probability of being in that state and multiply it with the 
probability of observing the sequence e from that state given the input sequence i'. The other parts 
can be explained analogously. 

Proposition 11. For all puts, s G S, d G D, q G Q, r G R, G I*, and e,e.£ E* , 

PT[[l(L,s)j(dd')\ E Ue] 

= u(x)Pv[ H(L,x)}(f)\E^e] where s A v 

Pr[ll(L,s)}(qJ)\ E Uq:J] 

= ^ "( x ) Pr [ ll(L, x)W)\ } where s^u 

x£S ± 

Pr[H(L,s)}(i)] E nr:(?] 

= v(x)Pr[ H(L,x)}(i)\e^] wheres^u 

X&Sj_ 

Proof. For the first equality of the proposition: Note that if e = [], then 

Pr[L[(L, S )l(^)J^e] = l= ]T v(x) Pr[ x)](?)Ji?3e] 

Otherwise, since s=i u, we know there exists \i such that s—tfi. It follows that 

PT[[l(L,s)}(d-A')\ E Ue]= Pr[ (23) 

a€7(e) 

= Pr[l(L,s)J(d-I)Ud:a!) (24) 

a' 67(e) 

= E E^(^x^V) ( 25 ) 

a'G7(e) s'gS 

= E EE^")*^ 5 ")^)^ 8 ') (26) 

W 67(e) s'GSs"es 
= Y u(x)Pr[[l(L,xm?)\ E m (27) 

x£S± 
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Line [M] follows since s — > fi implies that s does not transition under any outputs and e/[] implies 
that a € 7(e) cannot be []. Thus, we know that the first action of a must be of the form d:a! 
for Pr[ \{L, s)J(d:i')^\a] to be non-zero. Since \d:a!\E = e and d ^ E, [3'\e = e. Furthermore, 
last(a') = last(a) = last(e). Thus, a' G 7(e). Line [271 follows from Proposition PTOl 
For the second equality of the proposition: Note that if e* = [], then 

Pr[ll(L, S )}( q J)\ E ^q:J} = l= £ K*) Pr[ LK^K^k^ ] 

Otherwise, since s v, we know there exists \x such that s—tfi. It follows that 

Pr[ll(L,s)}(q:i>)\ E Uq:?]= E Pr[[(L, s)](g:?)I3a] (28) 

aG7(q:e') 

= J] PT[l(L,s)}( q J)^ q :a'} (29) 

= E E^ s )(^)(^ s ') ( 3 °) 

a' 67(5") s'GS 1 

= E E EM*><^">(?)(aV) (31) 

a'€7(e')s'eSs"eS 

= J2 ^)Pr[[l(L,x)j(i')\ E ^] (32) 

Line [29] follows since s4 fi implies that s does not transition under any outputs and (f ^ [ 
implies that a' G 7(e*) cannot be []. Thus, we know that the first action of a must be of the 
form q:a' for Pr[ [{L, s)](g:z')3a] to be non-zero. Since [q:a'\E = q'-e*, [S'\e = e 1 ■ Furthermore, 
last(a') = last(o) = last(e r ). Thus, a! G 7(e r ). Line 1321 follows from Proposition [TOl 
For the third equality of the proposition: Note that if e 1 ' = [], then 

Pv[[l{L,s)j(J)\ E Ur:e'] = l= £ !/(*) Pr[ L[(i^)l(?)J^e^] 

Otherwise, since s=>v, we know there exists \x such that s — > fx. It follows that 

PT[[l{L,8)l(t)\E^]= Yl Pr[{(L,s)J(T)Ua} (33) 

oG7(r:e) 

= ^ Pr[[(L, S )](?)3r:a'] (34) 
a'67(e') 

= E E^ s >(?)MV) ( 35 ) 

= E EE^^-O^^ (36) 

= £ K^)Pr[L[(^^)lWJ^] (37) 
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Line [3H follows since s4/i implies that s does not transition under any action other than r and 
e* 7^ [] implies that a' G 7(e y ) cannot be []. Thus, we know that the first action of a must be of 
the form r:a! for Pr[ \{L, s)J(i)Z|o] to be non-zero. Since [r:a?\E = rie 1 , [o?\e = e f - Furthermore, 
last(a') = last(a) = last(e"). Thus, a' G 7(e"). Line E3 follows from Proposition [lOj □ 

C Basic Properties of Differential Noninterference 

Sequence Differencing. Given the input sequences i\ and i 2 , A(ii,i 2 ) denotes the number of 
data points on which they differ: the minimum total number of data point insertions into i\ and 
i 2 it takes to make them equal. Formally, 

• A(ii,i 2 ) = iff i\ = i 2 . 

• For 1 < n, A(ii,«2) = n iff there exists d G D, i,i' 1} i 2 G /*, such that both of the following 
properties hold: 

— either %\ = i:d:i' x and i 2 = i-i 2 , or i\ = and i 2 = i:d:i 2 ; and 

- A(i[,i' 2 ) = n - 1. 

For A(i\,i 2 ) = n to hold for any n, ii and Z2 must agree on every query from Q: they may 
only differ by n data points from D. Since differential privacy is defined using data sets differing 
on one element, in most theorems we are interested in the case where A(ii,i 2 ) = 1, which means 
that there exists d G D, and G I* such that either i\ = i:d:i' and i 2 = , or i 2 = i:d:i' and 

For example, let d\ and d 2 range over elements in D, and q\ and q 2 range over elements in Q. 

• A([e?i, qi, d 2 ], \d\, q\\) = 1 (add d 2 to the end of the second sequence to get the first). 

• A([qi, d 2 ,q 2 ], [d\,qi,d 2 , q 2 }) = 1 (add d\ to the front of the first to get the second). 

• A([di, qi, d 2 , q 2 ], [d\, q±, q 2 ]) = 1 (add d 2 between q\ and q 2 of the second to get the first). 

• A([di, d 2 , qi, q 2 ], [dx, d 2 ,q 2 , qi]) is undefined (the two sequences do not agree on queries). 

Note that in the first example, the two sequences have a difference of one under the above definition 
but do not have a Hamming distance since they are of different lengths. 

While the choice of using all possible subsets of the set of trace prefixes instead of a single 
prefix makes the power of differential noninterference more apparent, it does not actually impose a 
stronger requirement as shown by the next lemma. This result simplifies reasoning about differential 
noninterference and is useful for proving subsequent results in this paper. 

Proposition 12. m has e- differential noninterference if and only if for all input sequences i\ and 
i 2 in I such that A(i\,i 2 ) < 1 and e in E* , 

Vi[[m(ii)\ E □ e] < exp(e) * Pv[[m(i 2 )\ E 3 e] 

Proof. The only if direction follows directly from the definition by setting S = {e}. 

For the if direction, arbitrarily fix i\ and i 2 such that A(«i, i 2 ) < 1 and S C E* . By assumption, 
for all e in E* , 

Pr[[m(ii)\ E □ e] < exp(e) * Pr[|_m(i* 2 )j£: 3 e] 
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Let S' be S with all the elements that are a longer version of another element of S removed. 
That is, S' = { e! G S \ $e G S s.t. □ e } where e" □ e means that e" is a strict prefix of e. Proof by 
induction over the length of e shows that for all e in S, there exists e* in S' such that e □ e'. Thus, 
if there exists e in 5 such that ["^iJJ-E 3 e, then there exists in S' such that L m (*i)j£ 3 e"- 
Thus, for all i, Pr[[m(i)\ E □ S] = Pr[Lm(i*)J £ □ 5']. 

For two and et> in 5' such that ^ et>, |_ m WJ-E can only have one of them as a prefix since 
neither is a prefix of the other. Thus, since S is countable, this implies that 

Pr[[m(i)\ E □ S] = Pr[[m(i)\ E □ ^] 
e"eS' 



Thus, 



Pr[L«i(i* 1 )J i3 □ 5] = Pr[Lm(?i)J E □ 5'] 

= Pr[Lm(ti)J B 3g'] 

e"eS' 

< ^ exp(e) * Pr[|_ra(i 2 )j£ 3 
= exp(e) ^ Pr[Lm(? 2 )J E □ e'] 

S'eS' 

= exp(e)Pr[Lm(? 2 )J B 3 S'} 
= exp(e)Pr[Lm(i 2 )j£; 3 S] 



□ 



The next theorem is analogous to previous results about differential privacy for functions: it 
proves that the privacy leakage bound for a system whose inputs differ on at most n data points 
is n * e where e is the leakage bound for the system if its inputs differ on one data point (see e.g., 
corollary of |MT07| ). 

Proposition 13. // a system m has e- differential noninterference, then for all input sequences i\ 
and i 2 such that A(ii,z 2 ) < n and for all S C E* , 

Pr[LlM](?i)J £ □ S] < exp(n * e) * Pr[L[M](? 2 )J B □ S] 

Proof. Proof by induction over n. 

Base Case: n = 0. In this case, i\ = i 2 and, thus, Pr[|_m(ii)j£: □ S] = Pr[|_m(i 2 )Je □ S] as 
needed with exp(0) = 1. 

Inductive Case: Assume for all n' < n; prove for n + 1. Since A(ii,i 2 ) = n + 1, there must 
exist z,^,^ G I* and d\ G D such that i\ = i:di:i'i, i 2 = i:i' 2 , and A(i' 1 ,i 2 ) = n. Let i% = i:z 2 :^. 
A(ii,^3) = 1 an d A(i3,i 2 ) = n. Thus, by the inductive hypothesis, 

Pr[Lm(?i)J B □ 5] < exp(l * e) * Pi[[m(i 3 )\ E □ 5] 

and 

Pi[[m(h)\ E □ S] < exp(n * e) * Pr[[m(t a )Ji5 3 5] 
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Thus, 



Pr[[m(ii)\ E □ 5] < exp(l * e) * (^exp(n * e) * Pr[[m(i 2 )j£ 3 5] J 
= exp(n + 1 * e) * Pr[|_m(i 2 )J.E 3 <S] 
as needed. □ 

D Compositional Reasoning 

To prove Theorem [H we use a definition and a proposition that helps us to track when the transition 
under h) is being simulated by many transitions of M 2 . 

Let Li = (Si,Qi Wi?i,->i) and L 2 = (S 2 ,tt,H 2 ,^ 2 ) and Af 2 = (^2,s 2 )- Let ^3 = 

Qi ttJ D\ ttJ i?i l±) ill t+J il 2 W {/i*}. Let h) be a distinguished internal action in H\. For simplicity, 
we assume that fv only labels the one transition of L\ that is implemented by M 2 . Let w be a 
distinguished internal action not in Hi or _ff 2 . 

Let ^(a) be a set of action sequences formed by replacing each action h' in a with the internal 
action h* followed by any sequence h from H^ and then h$ again. Formally, 

*(fc + :o) = h t :H^:h t :^(a) 

ty(a:a) = a:^(a) where a ^ 

where : is raised to work over sets in the standard way: for X C A* and K C X:Y = 
{ a G A* | 3ai € X, 3a 2 € Y s.t. a = a\\a 2 } and a:X = {a}:X. 

Proposition 14. Let M\ = (Li,sq) and let Ms = Mi[s' ,M 2 ,o] = (L^,sq) where M 2 implements 
the transition of fv under i. For all a, i, for all s £ S, 

s'eSi a'G*(a) s'eSi 

Proof. We use induction over the structure of a. 

Case: 2= [] ; Since *([]) = {[]}, (L u s) $)([], s') = (L 3 , s)(i)([], s') = 1. 
Case: a = i:a! . 

• Subcase: there does not exist i' such that i = i:i' and s — >i \i. In this subcase, (Li, s)(i)(a, s') = 
0. By definition of \E' we have that Va" G ^(o) = ^(i'.a?) and a" is of the form i:a!" for some 
a'". Since, by definition of —7-3, M3 has an input transition from a state only if it has an 

input transition from that same state in Mi there does not exist s — >3 [i. It follows that for 
all Va" G *(a), (L 3 , s)(i)(a", s') = 0, as needed. 

• Subcase: there does exist z' such that i = and s-\\ fi. In this subcase, 

(L 1 ,s)(i)(a,s') = fj, 1 (s")(L 1 ,s")(i)(a',s') where s-H/ii 

s"e5i 

and 

(L 3 ,s)(i)(a,s') = / u 2 (s")(L 3 ,s")(?)(a / ,s / ) where s4 3 ^ 2 . 
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Since each a" G ^ifl) is of the form i:a!" for some a'", we need to show that 

E E »i(s")(L 1 , s ")(?)tf,s')= E E E i»W(L a y)(?)(<t u ,s). 

s'eSi s"eSi i:3'"e*(3) s'eSi s"e5ittl5 2 

We reason as follows: 



E 


E 


E M^^'K^^V) 


(38) 


i:a"' £fy(a) 


s'eSi . 










= E E E (WX^V) 


(39) 














= E /*(*") E E^'H^V) 


(40) 






s"GSi i:a"'6*(S) s'eSi 








= E ^ s ") E E^'x^'V) 


(41) 






s"GSi o"'e*(a') s'eSi 








= E w( s ") E E^'x^'V) 


(42) 














= E E mi^o^i,^')^)^^') 


(43) 






s'eSi s"e5i 





Line [39] follows from reordering summations and using distributivity of multiplication over 
summation. Line HQ] follows from the fact that any s" G Supp(;U2) can not be in S2 since 
any s" G Supp(^2) is reachable via an input action. Line [UJ follows from, the fact that 
each a" G ^(a) is of the form i:a"' and a'" G '3/(3'). Line 02] follows from definition 
of —^3. We conclude in Line H3] using the inductive hypothesis Yls'eSi fii, s")(i')(a', s') = 

Case: a = o:a! . 
• Subcase: o/ftt. 

— Subsubcase: there does not exist a! such that a = o:a! and s — >i \x. In this subcase, 
(Li, s)(i)(a, s') = 0. By case definition we know a = o:a' where o G i?i tt) H\ and o^w . 
Then, Va" G ^(a) = 5 r (o:a / ), a" is of the form o:af" for some a'". Since, by definition 
of —7-3, M3 has an output transition on an action from R± fcfcl H\ \{fv} only if it has the 
same transition M\. This gives (L3, s)(i)(a", s') = for all Va" G *(a), as needed. 

— Subsubcase: there does exist a' such that a = o:a! and s-?i /x. In this subcase, the proof 
follows a line of reasoning analogous to the case where a = i:ct. We show that 

E E Vi(s")(L 1 ,s")(j)(a',s')= E E E Ms")(L 3 ,s")(i)(a!",s'). 
s'eSi s"eSi o:S'"e*(a) s'eSi s"GS , itus 2 

In the step where we argue that s" ^ S2, we use the fact that all states in £2 can only 
result from a transition on or an action from H2, and that o G R\ Lfc) -f/i \ {/i^}, which 
is disjoint from {hs} l±) i?2- 
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• Subcase: o = fv . 

— Subsubcase: there does not exist \i such that s \\ /i, then (L±, s)(i)(a, s') = and s ^ . 
In this case all a" £ Vf(a) start with h$hh$ for some vector h of actions from '. Since 
s , according to the definition of —7-3 the only way for s to have a transition on 

h} is if s = i(si) for some s\ E Supp(/i^), where \\ yfi and that transition is 

t(si) —7-3 Dirac(si). By definition of —7-3, si can only transition on actions present in Mi, 
which means it cannot transition on any actions from H2. This makes it impossible for 
h} to be followed by a sequence h from actions H2, and gives (L3, s)(i)(a", s') = for all 
a" G ^(a), as needed. 

— Subsubcase: there does exist fi such that s Ai /i, then by the assumption that is the 

unique state enabling h) we know that s = and by transition-determinism A/A 
We need to show that 



£ /*V)(£i , s")® (*V) = Yl £<W>(?)(3V)- 

s'GSi s"eSi a"G*(a) s'eSi 

We reason as follows: 

]T £(L 3 , S t)(?)(3" )S ') (44) 

a"g*(a) s'GSi 

E £<W>(^:/^:a'V) (45) 

ht:h:ht:a'"e^(a) s '^ s i 

E £<L 3 ,^>W(^:a'V) (46) 

ht:K:/it ea "'e*(a) 

E E E ^3,^)([])(/J, S " , )*^3, S , ")(?)(^:a / ", S / ) (47) 

/lt:£:htS"'etf(3) s ' 6S i s"'eSittJ& 

E EE <i S ,«s>([])(^o*<A.,o(5)(fc t :a w y) (48) 
E EE (i3.'S>([])fto*(^3.' 4 xs)(a w y) (49) 

ht:h:hta'"e*(a) s"'et(St) 

where s"'A 3 Dirac(s 4 ) (50) 

= E ( E (^xnxM'"))* E E (^yx^V) (51) 

s"'et(5t) he(H 2 )+ s ' eS, i 3"'e*(S') 

where s"'A 3 Dirac(s 4 ) (52) 

= E mV)£ E (^X^'V) (53) 
s'"ei(St) s'eSi a"'ey(a') 

For Lines 33] to H7] we observe that by definition of each a" 6 \& (a) is of the form 
h^:h:h$:a"' where h ^ [], by definition of —7-3, A3 Dirac^) and use Proposition [6j 
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For Line 08] let t(S^) = {s G L2 | s = t(s') for some s' G Supp(/i^)}. The equation 
follows since (L3, s^KHX^ = for all s'" G Si, by the assumption that for all 
s G Supp(/xt), jU'(s) = X^hg(_H- 2 )+ -^2([])(^j and that /i is a probability distribution. 

By definition of — ^3 we know that t(si) A3 Dirac(si) for all si G Supp(//t) and Line 1501 
follows. Note that by definition of —7-3 and i(S') where since 1 is an injection we know 
that for each s'" G l(S^) there is a unique state s 4 G Supp(//t) such that s'" = t(s 4 ). By 
using distributivity of multiplication over addition and the fact that a'" G ^f(a') we get 
Line[52j Line 1531 follows from the assumption that for all s G Supp(/i^), 

he(H')+ 

To conclude this case we recall that for each s" 1 G t(<S 1 ') there is a unique state s 4 G 
Supp(/^) such that s'" = t(s 4 ) and use the inductive hypothesis. 

□ 



Proof of Theorem [TJ Let M x = (Ia,s ) and let M 3 = M 1 [s\M 2 ,l) = (L 3 ,s ). We show that 
for all i in I*, and e in E*, s in Si, Pr[ \_(Li,s)(i)\e 3 e ] = Pr[ L(-^3> s )wJ-E 3 e]. By expanding 
the definitions of Pr[ [(Li, s)(i)\e 3 e] and Pr[ [(^3, s)(i)Js □ e] we get 

Pr[L(Li,s)(?)J B 3e]= 2 Pr[ (Li, s)(?) □ e] 

0671(g) 

= E E^' s )( ? )( 3 ' s ') 

aS7i(e) s'eSi 

and 

Pr[L(L 3)S )(i')J S 3e]= J] Pr[(L 3 , S )(?)3e1 

aG73 (e) 

= E E ^3, S )W(a, S ') 

0673(e) s'eSiwSi 

where 71 and 73 are sets of sequences of actions, respectively, of Mi and M3 defined as follows: 
7l ifi) = I IaI^i = e A last(a) = last(e) } with the special case that 7i([]) = {[]}, and 

73(e) = { a G A3 I L^Js 3 = e A last(a) = last(e) } with the special case that 73([]) = {[]} (as justified 
at the end of Appendix [B]). Note that we use A{ for the set of all actions of Mj and £j for the set 
of observable actions of Mj. 
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Now we show that for all s G Si, i, e, 

E ^(L 1)S )(?)(a, S ')= E E E< L 3> S >( ? X«V) (54) 

a67i(e)s'65i 0671 (e) 3'6*(a) s'eSi 

= E E E <^3, *}(?)(«>') (55) 

0671(6) a'6*(a) s'GSiliJ5 2 

= E E ^3, ^(a,^) (56) 

a£$(e) s'e5il+)5 2 

= E E <^3, *>(?)(«,*') (57) 

a£7 3 (e) s'eSittiSa 

where <3?(e) = Uae7i(e) ^(^)> an< ^ ^ * s as defined at the top of this section. 
Line [52] follows from Proposition [TU 

For Line l55| we argue as follows: Since M<i has no external actions and all transitions of M3 
on external actions end in a state in Si \ S2, for those states s' S S2, s' is reachable via a hidden 
action only. Thus, for any i, any a E 73(e), (L3, s)(i)(a, s') = since a ends in an observable action 
from E by definition of 73. 

Line [56] follows from the definition of 3> and the fact that for any pair of sequences d[ , 02 such 
that d[ 7^ 02, *(al) n ^(02) = 0. 

For Line [57] we observe that any sequence a E 73(e) \<J>(e) must have an occurrence of the action 
h$ that is neither immediately preceded by a subsequence of the form h$:h or immediately followed 
by a subsequence of the form h:h*. Then, by definition of —^3, (L3, s)(i)(a, s') = for all sequences 
a G 73(e) \ $(e), giving the needed equation. 

E Proof of Soundness of Unwinding 
E.l A Helpful Proposition 

Proposition 15. If (3 is a bijection from Supp(z^i) toSupp(^) and for all x'^ G Supp(^i) ; |lnz/i(a^)— 
lrn/ 2 (/3(a?i))l < *i then 

E ^i(xi)ex P (e'- ( J)Pr[L[(L^(x' 1 ))](?)J £ 3e'] 
x' 1 eSupp(i/i) 

<exp(e') E V2(4)Pr[ll(L,x 2 )J®\ E ^] 

x' 2 eS± 

Proof. For all x\ in Supp(z^i), 



exp(e' -S) = J^sM Wi)) ex P( e ' " 5 ) ( 58 ) 

= exp(ln(^ 1 (x , 1 )) - In(^G8(xi))))^G8(xi)) exp(e' - 5) (59) 

= exp(e , - 5 + ln(^K)) - ln(z/ 2 (/3K))))z/ 2 (/?K)) (60) 

< exp(e')^ 2 (/3(x , 1 )) (61) 
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Line 1591 follows from the fact that for every x[ G Supp(^i), 

viWfaMx'J) = exp(ln( Z , 1 K)))/exp(ln(^ 2 (/3(x / 1 )))) 

Line EH follows since for every x[ G Supp(i/i), | Yn.v\(x']) — In v 2 ) ) I — & implies that Irvviix'^) — 
lniAj(/3(xi)) <<5. 
Thus, 

2 Z / 1 (r B / 1 )ex P (e / -«5)Pr[L[(L,/3(x' 1 ))](?)j E 3 e -'] (62) 

x^€Supp(fi) 

< Y, exp(e') I / 2 (^(^))Pr[L[(L,/3(xi))](?)J B 3^] (63) 
x' 1 eSupp(^i) 

= £ exp( e > 2 (4)Pr[L[<L,4>K*)Ji^] (64) 
x-^eSupp(i/ 2 ) 

= exp(e') Y, V2(x' 2 )Pr[ll(L,x' 2 m\E^] (65) 

Line [64] follows from the fact that f3 is a bijection from Supp(^i) to Supp(zv 2 ). □ 
E.2 Proof of Lemma [1] 

Below we prove that Pr[ \_l(L, xi)](«)J.El3e] < exp(e) Pr[ [{(L, x 2 )J(J)\e^^]- Proving the reverse 
that Pr[ H(L, x 2 )J(i)\E^e] < exp(e)Pr[ [{{L, xi}J(i) \ E~^e] is much the same reversing the roles of 
x\ and X2 and using /3 _1 in the place of (3. 

Proof by induction over the structures of e and i. 

Case: e = []. In this case, 

Pr[ [l(L,x 1 )j(i)\ E □[]] = !< exp(e) * 1 = exp(e)Pr[ [l(L,x 2 )j(i)\ E □ []] 

Case: x\ has no outgoing transitions and e / []. In this case, Pr[ [{(L, xi)](i)J_e3e] = < 
exp(6)Pr[LI(L,x 2 >](?)j^]. 

Henceforth, we only consider 27 with at least one out going transition. Since x\ is related to 
a?2> we know it must also have at least one out going transition. Thus, neither x\ nor x 2 can be _L 
Thus, we use s\ for x\ and s 2 for x 2 for the reminder of the proof. 

Case: e = (7:6* and i = [] for some q £ Q. In this case, 

Pr[LI(L, Sl )K?)J E 3g:e- / ]= £ P*[[<A*i>](S)3<f] = £ £ <L, Sl )(i)(a, S ' x ) 

Since e / [], [] is not in 7(e). Furthermore, all a in 7(e) must have q come before any other action of 
E. In particular, a must have either the form q:a! , <i:a', or /i:a' for some aG#,(lGD, and h € H. 
Since s\ is //-disabled by being in the unwinding relation, we know that for no h £ H and fi does 

81 A/i. These factors combine to mean that (L, si)([])(a, s^) = for all G 5 and a G 7(e). Thus, 
Pr[ UK-Mi)] ([DJe^J":^] = 0. The same reasoning concludes that Pr[ [{{L, s 2 )j([])\ E Uq-e' } = 
making Pr[H(L, Sl )]([})\ E Uq:? } < exp(e) Pr[ [{(L, s 2 )}([])\ E ^q^] since < exp(e)0 
Case: e = r:e! and % = [] for some r G R. We consider the following subcases: 
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Subcase: s±^fi for some fi. Since s\ and S2 are related, there exists ^2 such that S2 — > ^2- 
This implies that there exists v\ and V2 such that si=^z7 and 52=^^2- Since s\ 7£ e S2, 
there exists S in [0, e] such that v\ £(lZ e ~ s ,5) v<i. This implies there exists a bijection j3 
from Supp(z^i) to Supp(^2) such that for all x\ € Supp(z^i), xilZ e ~ s (3(x2) and |lni/i(cci) — 
In i/2(P(xi))\ < 5. Thus, we may apply the inductive hypothesis to i and e 1 to get for all x\ 
in Supp(i^), Pr[ limx'Ml)^} < exp(e - <5)Pr[ [{(L, P(x[))}(i)\ E U? }■ Thus, 

Pr [ ll(L,s 1 )}(i)\ E ^r:e'} (66) 
= £ vxix'JPvllKL^'Mi)]^} (67) 

a^GSx 

= y, M^'iWumx'M^E^] (68) 

x' 1 eSupp(i/i) 

< Yl ^iK)exp(e-5)Pr[L[(L,/3(xi))l(?)J f; 3e'] (69) 
x' 1 eSupp(i/i) 

<exp(e) £ z, 2 (x 2 )Pr[L[(L,x 2 )](i)J^ e -'] (70) 

x^GSx 

= exp(e)Pr[L[(L, S2 )l«J^e- / ] (71) 

Lines [67] and [71] follow from Proposition [TT] Line [69] follows from the inductive hypothesis 
Line [70] follows Proposition PT5l 

r' 

Subcase: si — > \x for some output r / r. Since si and S2 are related, there exists \i2 such that 
■52 -> /"2- Furthermore, for no other action a ^ r' does does si A // or S2 A /Li' for any //. Recall 
thatPr[L[(L, S i)J(?)J E 3r^ E^es^* 
For all a £ 7(r:e ) '), its first element from .E must be r and, thus, it cannot start with r'. How- 
ever, s can only transition under r' and 5 / [], meaning there must be a transition for a to be 
produced. Thus, for all such a and s[, (L, si)(i)(a, s[) = and Pr[ [^{L, si)J(T)\E^r:e'] = 0. 
Similar reasoning concludes that Pr[ [{(L, S2)}(i)\E^r:(? ] = 0. Thus, Pr[ [{{L, si)}(i)\E^r:^ } 
< exp(e) * = exp(e) Pr[ [{(L, S2)\{i)\E^'r:e l ] as needed. 

Subcase: s\ is an input accepting state and i = []. Recall that 

Pr[L[(L, Sl >K*)k3r:e<]= £ £ (L, 8l )(t)(S, s[) 

ae7(r:e») s^GS 

Since a cannot be [], i = [], and s\ is an input accepting state, this means that {L, si)(i)(a, s^) - 
for all such a and s[. Thus, Pr[ [{{L, s 1 )}(i)\ E ^r:&] = 0. 

Since s\ is input accepting and related to S2, S2 must also be input accepting. Thus, by 
similar reasoning Pr[ U(L, <si)J(i)j£;3r:e 1 '] = and the results holds as above. 

Subcase: s\ is an input accepting state and i = q:i for some q G Q. Since e = r:e", no 
a E 7(e) can have q come before r. Thus, much as above Pr[ [l_(L, si)}(i)\E^r:(^ ] = = 
Pv[[l(L, Sl )}(i)\EUr^}. 
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• Subcase: s± is an input accepting state and i = d:i! for some d G D. Since s\ is in- 
put accepting and related to S2, «2 must also be input accepting. Thus, there exist v\ 
and V2 such that s\ =^ v\ and 52=^^2- Since s\ TZ e S2, there exists 5 in [0,e] such that 
v\ £(7Z e ~ s ,5) V2- This implies there exists a bijection f3 from Supp(z/i) to Supp(^2) such that 
for all x\ G Supp(i/i), x\7l e ~ s (3(x2) and | In z^i(xi) — In U2(f3(xi))\ < 5. Thus, we may apply the 
inductive hypothesis to i' and r:e" to get for all xi in Supp(Vi), Pr[ [J(£j ^'i )](*') J -E^^^] < 
exp(e - «S)Pr[ LI^teM?).!^^]. Thus, 

Pr[LI(L, Sl )Kd:?)J^^:e ] (72) 
= Y M4mWM(?)}E3T:i*] (73) 

= £ ^(^PrlLKA^K?)]^^] (74) 
x'' 1 eSupp(i/i) 

< £ ^ 1 (x' 1 )exp(e-«5)Pr[L[(L,/?(x' 1 ))K?)J^^] (75) 

a-' 1 eSupp(^i) 

<exp(e) ^ ^(^PrtLKL.^K?)]^:^] (76) 

= exp(e) Pr[ [{(L, s 2 )}(d-J)\ E ^r:J ] (77) 

Lines [73] and [77] follow from Proposition [TT] Line [75] follows from the inductive hypothesis 
Line [76] follows Proposition [15] 

Case: e = q:e* and i = for some i in I and i in I*. We consider the following subcases: 

• Subcase: s\ is not an input accepting state: there exists no fj,\ such that si Since si and 
S2 are related, there also cannot exist a /i2 such that s 2 -^- ^2- Since s± does have a transition 
and is //-disabled, there must exist some response r such that si A ^ and S2 — > H 2 f° r some 

and fj,' 2 . Furthermore, si and S2 transitions under no other actions. Recall that 

Pr[H(L,s 1 )}(iJ)] E nq;?)= £ Pr[ {(L, Sl )J(i-I)Ua) 

= E *!>(*?)(*, *i) 

aey(g:e") 

For all a G ^(q-.e 1 ), its first element from I? must be q and, thus, it cannot start with r. How- 
ever, s can only transition under r and o / [], meaning there must be a transition for a to be 
produced. Thus, for all such a and s[, (L, si)(i)(a, s[) = and Pr[ [{(L, s\)\{i:i')\El^q'-^ ] = 0. 
Similar reasoning concludes that Pr[ \_{(L, s 2 )}(i'-i')\E^q-e > ] = 0. Thus, 

Pr[ ll{L,si)}(i:i')\ E Uq--e] < exp(e) * = exp(e) Pr[ H(L,s 2 )J(qJ)] E n q :g] 

as needed. 

• Subcase: s\ is an input accepting state and i = q for some n\. Since s\ is input accepting, 
si-^Hi for some [X\. Since s\ and S2 are related, there exists ji 2 such that S2-^n 2 - This 
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implies that there exists z^i and f 2 such that s\ =>- v\ and s 2 => v-i- Since s\ lZ e S2, there exists 
5 in [0, e] such that v\ C(lZ e ~ s ,5) ^ 2 . This implies there exists a bijection (3 from Supp(^i) 
to Supp(z/ 2 ) such that for all x\ G Supp(z^i), x{R, e ~ 5 f3{x2) and | \n.v\{x\) — h\V2{p{x\j)\ < S. 
Thus, we may apply the inductive hypothesis to i' and e! to get for all x\ in Supp(fi), 
Pr[L[(i,^i>](?)J^] <exp(e-5)Pr[L[(L,/3( 2 ;' 1 ))](?)J £ ;3e"]. Thus, 

PrtLK^^Kg:?)]^:^] (78) 

= £ ^(x'jPrfLKL^DK?)]^] (79) 

a;iGS x 

= vAAWYI^A)\$)\e^\ (so) 

x-' 1 eSupp(i/i) 

< ^iK)exp(6-,5)Pr[L[(L,/3K))K?)J^e-'] (81) 

x' 1 eSupp(i/i) 

<exp(e) £ i, 2 (4)Pr[L[(L,4)K?)J^] (82) 

= exp(e) Pr[ L[(L, a^K^Ji^ ] (83) 

Lines [79] and [83] follow from Proposition [TT] Line [81] follows from the inductive hypothesis 
Line [S5] follows Proposition [TS] 

• Subcase: si is input accepting, i ^ q, and i £ Q. Recall that 

Pr[H(L,s 1 )}(i:i , )] E nq;?)= ]T Pr[ {(L, Sl )J(i:l)Ua) 

367(g:e ? ) 
aS7((j:e') s^eS 

For all a G 7(q:e 1 '), its first element from 12 must be q and, thus, it cannot start with 
i. Thus, for all such a and s' l7 (L, si)(i:i')(a, s^) = and Pr[ [^{L, s^^r.i^lE^q-e 1 ] = 
0. Similar reasoning allows us to conclude that Pr[ [{(L, S2)}(i-i')\E^Q'-e' ] = 0. Thus, 
Pr [ H(L, si)] (i:?)J e^ie* ] = < exp(e) * = exp(e) Pr [ fl 2 )l (q$)\ E^q:? } as needed. 

• Subcase: si is input accepting, i ^ q, and i £ D. We use d to denote i. Since si is 
input accepting and related to s 2 , s 2 must also be input accepting. Thus, there exist v\ 

and V2 such that s\ =i v\ and s 2 4>^ 2 . Since si 1Z e s 2 , there exists 5 in [0, e] such that 
v\ C{lZ e ~ 5 ,5) V2- This implies there exists a bijection /3 from Supp(z/i) to Supp(f 2 ) such that 
for all a; 1 € Supp(z^i), x\R. e ~ s f3{x2) and | \xlv\(x{) — In z^ 2 (/3(xi))| < <5. Thus, we may apply the 
inductive hypothesis to and q:e* to get for all x\ in Supp(^i), Pr[ L|(L, Xj)J(i )Js3g:^] < 
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exp( e - 6)Pr[ L[(L,/3(xi))](?')J^g^]. Thus, 
Pr[L|[(L,s 1 )](d:?)J^g:l'] 

= £ ^ 1 (x' 1 )Pr[L[(^,x' 1 )K?)J^e- / ] 
x'' 1 eSupp(i/i) 

< Yl ^ ex P( £ " <*) Pr t LI<^. /3(*'i))K?)k^ 
x' 1 eSupp(^i) 

<exp(e) ]T Mx' 2 )Pt[[1(L,x' 2 )}^)\e^] 
x' 2 eS± 

= exp(e)PT[ll(L,s 2 )j(d-A)\ E Uq:g) 



84 
85 

86 
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Lines [85] and [89] follow from Proposition [TT] Line [87] follows from the inductive hypothesis 
Line [88] follows Proposition [15] 

E.3 Proof of Theorem H 

We use Lemma [T2l and strengthen the hypothesis to show that for all reachable states s and e, 
Pv[[l(L,s)}(i 1 )\ E ne]= Y Pr[l(L,s)}(hU3] 

aG7(e) 

<exp(e) ]T Pr[l(L,s)J(i 2 )^a] 

= e X p(e)Pr[L[(L, S )](? 2 )J s 3e] 

Arbitrarily fix i\ and i 2 such that A(ii,22) = 1. We use induction over the structures of i\, i 2 , and 
e. 

Case: e = []. In this case, 7(e) = {[]} and Pr[ [{{L, s)K«i)Jb3[] ] = 1 < exp(e) * 1 = 
exp(e) Pr[ [{(L, s)1(i 2 ) \ e~^[] } irrespective of i\ and i 2 . 

Only in the case where e = [], can [] be in 7(e). Thus, we assume that e 7^ [] in the reminder 
of this proof. 

Case: n = [} and i 2 = []. Pr[ f(L, s)J ([])□«] = Pr[ [(L, s)I([])3o] for all a £ 7(e) for any e. 
Case: zi = (i:^ and £2 = d:i 2 . We consider three mutually exclusive subcases: 

• Subcase: sA-fi. For all a such that for no af, a = d:af, Pr[ [(L, s)](«i)3«] = = Pr[ {(L, s)}(i 2 )^a] 
Since such a add nothing to the summations, we may ignore them and limit our attention to 
a = d:a' in 7(e). Note that all such a' are in 7(e) iff d:a! is in 7(e). 

All the states in Supp( / u) are reachable. Thus, for each state s' in Supp(/x), we may apply the 
inductive hypothesis on i' 2 , and e to get that 

]T Pr[[(L, S ')l(?i)^]<exp(e) £ Pr[ {(L, ] 
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Considering the sum over all such a, we get 

£ Pr[ [<£,*>]&)□*] (90) 

aS7(e) 

= Yl Pr[l(L,s)J(i 1 )^d:a'} (91) 

= E EM s, )Pr[[(^ S ')l(?i)^] (92) 
3' 67(e) s'e5 

= E^ S ') E PrtlMK^ef] (93) 

s'GS a'ey(e) 

= E M*') E Pr[[(L, ,')](?;)□«'] (94) 

s'eSupp(fi) a'G7(e) 

< J] Ms')exp(e) £ Pr[[(L, S ')K? 2 )3«'] (95) 

s'eSupp(/^) a'G7(e) 

= exp(e) £ ^^OPrfKL,^)]^)^] (96) 

W 67(e) s'6S 

= exp(e) ^ Pr[[(L, S )Ktad:a'] (97) 

d:a' 67(e) 

= exp(e) Pr[I(L, S )l(i 2 pa] (98) 

aG7(e) 

where d:a! in the expression <i:a' G 7(e) ranges over only those elements of 7(e) of the form 
d:a! . That is, J2d-d'€y(e) * s shorthand for 

£ 

d:a'e{a"ej(e) \ 3a' £A* s.t. d:a'=a" } 

Note that the last line follows from the fact that [(L, s)](i2)(a) = for all a not of the form 
d:a'. Lines [92] and [97] follow from Proposition [9] Line [95] follows from the inductive hypothesis. 

• Subcase: s4/j for some r£iJ. For all a such that for no a' , a = r:a', Pr[ [(L, = 
= Pr[ |(L, s)](«2)3a]- Since such a add nothing to the summations, we may ignore them 
and limit our attention to r:a! in 7(e). Unless e = rre* for some e* , no such r:a' will be in 7(e) 
and both summations will be zero. Thus, we limit our attention to the case where e = r:e / 
for some e! . In this case, we may use the inductive hypothesis on i±, 12, and e! to get that for 
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all s'GSuppOi), E^^PrtK^^Kn)^^] <exp(e)Ea' 67(e -)Pr[[(L, S / )K?2)3a']. Thus, 



E Pr t 

a€7(e) 


[(L, S )](?i)3a] 


(99) 




= £ Pr[[(L, S )K?i)3r:a'] 

r:a'(E7(r:?) 


(100) 




= E E^ s ')Pr[[(^, S ')Kn)3«'] 


(101) 




= E MOO E Pr[[(i, 5 ')Kn)3a] 

s'eSupp(/x) a'£7(e*) 


(102) 




< £ M(*0«p(e) E P*[l(L,s')}(i 2 )^a?] 

s'eSupp(/x) a'G7(e') 


(103) 




= e X p(e) £ ^/x( S ')Pr[[(L jS , )l(4)33 / ] 


(104) 




= e X p(e) ^ Pr[[(L,s)](i 2 )3r:3'] 

r:a'67(r:f?) 


(105) 




= exp(e) ^ Pr[[(L, S ) J (?,)□<?] 


(106) 



a€7(e) 



Lines [TOTl and [T05l follow from Proposition [9j Line 11031 follows from the inductive hypothesis. 

• Subcase: Otherwise. Since s is //-disabled, it is not the case that s fi for any fi or h G iZ\ 
Since a / [], Pr[ [(£, a)](ii)3a] = = Pr[ [(L, s)J(? 2 )3a] for all a G 7(e). 

Case: ij = q:ii and & 2 = <7 : * 2 - Much as above just using that a is only in 7(e) if e = q:e* for 
some e" and a' G 7(e*). 

Case: i 2 = rf:ii . We consider the following subcases: 

• Subcase: s—>fi. Since s—>(i, for some v, s=izA Since s is reachable from sq, there exists an 
e-unwinding relation 1Z e that covers s and d. That is, for all s' G Supp(^), for all s' G Supp(f), 
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s Tl e s' and v(±) = 0. 



PrMIYL s)ll 1 rp(ii)Zie] < exu(e) 

xi LLI[\^i' , /JIJ-C'V''l/ — C J — CA FV C J 


Pr[ 1 l(L s m -,„)i 1 r?(u)Zie] 


(107) 




1 v(x) 1 Prf 1 UL s^-, n )}(u) 1 pZlel 


(1081 


= exp(e) 


/ \ 

2>(*0 Pr[LI(L, S min)l(?i)J E 3e] 
\s'es J 


(109) 


< exp(e) 


^2u( S , )Pr[l[(L, S , )l(t 1 )\ E ^e] 


(110) 


= exp(e) 




(111) 


= exp(e) 


Pr[l[{L )S )}(d^)\ E ^ 


(112) 


= exp(e) 


PT[[l(L,s)j(i 2 )] E Ue] 


(113) 



where s m \ n is the state s' S Supp(z/) that minimizes Pr[ H(L, s')}(ii)\ E ^e]. Line 11071 follows 
from Lemma[TJ Line 1 1081 follows from Proposition [71 Lines [T09l and [TTT1 follow from ^(-L) = 0. 
Line 1 1 1 2l follows from Proposition 1111 

• Subcase: s4/i for some r. As in the corresponding subcase in the case for i\ = d:i\ and 
i2 = d:i' 2 , we may ignore a not of the form a = r:a' and e not of the form r:e". In this case, 
we may use the inductive hypothesis on i\, i 2 , and e* as before to get the required result. 

• Subcase: Otherwise. Since s does not transition under d in this case and the automaton has 
quasi-input enabling, it does not transition under any input action. Further, s is ii-disabled. 
Thus, since a/ [], Pr[[(L,s)](ti)Do] = for all a £ 7(e). 

Case: ii = d:i2- We consider the following subcases. 

• Subcase: s— Since s—>fi, for some u, sA>v. Since s is reachable from sq, there exists an 
e-unwinding relation 1Z £ that covers s and d. That is, for all s' € Supp(i^), for all s' £ Supp(z^), 
s 7?. e s' and i/(_L) = 0. 

Thus, 

Pr[L[(L, S )](i* 1 )j^e] = Pr[L[(L, S )K^ 2 )J^e] (114) 
= ^ K*)Pr[L[<L,*>](? 2 )j^e] (115) 

= ^ Kx)Pr[L[(L,x)K? 2 )J^e] (116) 
xeSuppjV) 

< J] z,(x)exp( e )Pr[L[(L, S )K? 2 )J^e] (117) 

a;eSupp(i/) 

2 !/(*) J exp(e)Pr[L[(L, S )K? 2 )J^e] (118) 

^i'GSupp(^) y 

exp(e)Pr[L[(L, S )K? 2 )J^e] (119) 
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Line II 151 follows from Proposition [TTJ Line 11171 follows from Lemma [TJ Line II 191 follows from 
Proposition [7J 

• Subcase: s4/i for some r. As above in the other subcases for s4/i. 

• Subcase: Otherwise. In the case where for no fj, and a / [], everything is 0, which is 
lower than any possible value of exp(e) Pr[ H(L, s)](«2)J_B3e]. 

F Proof of Lemma [2t M ex i(K) has an Unwinding Family 

To prove Lemma [2J arbitrarily fix a state s and data point d. We use proof by induction over j 
from to t to show that for each pair of states s\ and S2 such that si Tl 2 /^ S2, they have the needed 
properties. 

In both the base or inductive cases, since s\ S2, S2 must have the same value for the PC as 

s\. Thus, they have the same set of enabled actions. That is, there exists a ui such that s\ A ui 
iff there exists a /i2 such that S2 — > fJ-2- Thus, s± =^ ui iff S2 => i>2- 

Base Case: j = 0. For states with a PC of 08, the properties follows from the related states 
being equal. 

For states with a PC of 16, we can prove the needed properties using 5 = as we must since 
1Z° ad is a 0-unwinding relation. Since j = and si Tl 2 J d S2, si must have the form 
(16, {B' , . . . (no, . . . , n' t _i), d, y', r', k'). Since s± is related to another state, it must be in S{. 

Thus, si is reachable in t queries and d = c+ (t — 1). Once curSlot is updated by line 17, it will roll 

over to the value of c. Thus, s± ^> Dirac(s' 1 ) where s[ = (08, {Bq , . . . , B"_ x ), (n ', . . . , n"_i), c, y' , r', k')) 
where B c = •{[]}, n" = 0, and for all c" ^ c, B" c „ = B' c „ and n!' c „ = n' c „. Since the cth slot was 
holding the data point by which s± and add(si, c, d) differ and s\ differs from swap(si, c, d, d') for 

each value of d', add(si, c, d) =4> Dirac(s' 1 ) and swap(si, c, d, d ) Dirac(s 1 ) for all d! . We use (3 that 
maps s'y to itself and nothing else to anything. Furthermore, for the one state in Supp(^i), 
| In i^i (s^) — In (/9(s^ ) ) | =0 = 5. Thus, v\ £(=,0) U2 where equality is trivially a 0-unwinding 
relation. 

Inductive Case: j > 0. We consider cases depending on what type of action a is to show that 
there exists 5 in [0,2je] such that £(7Z 2 J^~ S ,8) fi2- 

• Subcase: a € D. In this case, we prove that such a 5 exists using 5 = 0. That is, we prove 
that ui CilZ 2 ?^ 0) V2- Since a G D, s\ has must have the form 

(08, (B' Q , ... , B'^), (n' , . . . nf^), d, y', r', k') 
We consider subsub cases: 

— Subsubcase: c = d and n c > < v — 1. In this case, both states s\ and S2 will store 
the data point a. For that d, v x = Dirac((08, B", ft", d, a, r', k')) where B" d = B' d tt) 
{a}, n", = n' c , + 1, and for all c" / c', B" c „ = B' c „ and n"„ = n' c „. Similarly, z/2 = 
Dirac((08, (B"',n'", d, a, r' , k')) where either 
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1. B' d ' = B c ,to{d}to{a}, n% = n' c , + 2, and for all c^c" ^ d, B'jj, = B' d , and </, = <„; 
or 

2. B' d ' = B d to {d} - {d'} W {a}, </ = n' c , + 2, and for all c^d' ^d, B' d , = B' c „ and 
n% = n' c „ for some d' . 

Thus, for s[ G Supp(z^i) and s 2 G Supp(i/ 2 ), s' 2 is either add(s' l5 c, d) or swap(s^, c, d, d') 
for some d'. 

To show that fi\ C(TZ^ d ,0) fi2, we use the function (3 that maps s' x to the state s' 2 
and nothing else. Since both v\ and f 2 are Dirac distributions, that covers all of their 
supports and is a bijection. It follows from s' 2 being either add(s' 1 , c, d) of swap(s' 1 , c, <i, d') 
for some cf that s[ 7£^J s' 2 . Lastly, | In z^i ( ) — lnz^ 2 (,s 2 )| = | In 1 — In 1| =0 < 5 

— Subsubcase: d and n c i < v. Mostly, as above. 

— Subsubcase: n c i = v. In this case, both states s\ and s 2 will drop the data point a and 
not store it. For that d, v\ = Dirac(si) and v 2 = Dirac(s 2 ) By assumption, s\ TZ 2 /^ s 2 . 
| ln^i(si) - lnu 2 (s 2 )\ = jlnl -lnl| = < 8 

— Subsubcase: c = d and n c i = v — 1. If s 2 = swap(si, c, d, d') for some d', then this 
subsubcase is the same as the first one. Otherwise, the si will store the data point, 
but s 2 = add(si,c, d) will not since it already has n c i + 1 = v data points. Thus, v 2 = 
Dirac(s 2 ) and v x = Dirac(s' 1 ) where s' x = (08, (B%, B"_^), (n'{, . . . , n^), c', a, r', k')) 
where B" d = B' c , ttl {a}, n' d = n' d + 1, and for all c" / d, B' d , = B' c „ and n d , = n' c „. Thus, 
we have that s 2 = swap(s' 1 , c, d, a). Thus, «2- We use /3 that maps to s 2 and 
nothing else. Since |lnz/i(si) — lni/ 2 (s 2 )| = 0, v\ C(TZ 2 J d d ,,0) v 2 . 

• Subcase: a G -R. In this case, we prove that such a 5 exists using 5 = 0. That is, we prove 
that v x £(^J,0) i/ 2 . 

Since a £ R, s\ must have the form (16, (-B , . . . , B' t _]), (no, . . . , n't-i), d, y' , r', k'). Thus, 
s± =>- Dirac(s' 1 ) where 

si = (08,(^ , ,..., J Bti),K , ,... ) <_i),c / + l mod t,y',r',k')) 

where B c+1 modt = {{}}, < +1 mod t = 0, and for all d' ^ c + 1 modi, fl» = and 
<„ =n' c „. 

If s 2 = add(si, c,d), then s 2 =4> Dirac(s 2 ) where 

s 2 = (16, (JBq, . . .,B"_ 1 ),(n , Q,...n"_ 1 )d + 1 mod t,y',r',k')) 

where B^ +1 mod t = {{}}, J3£ = B' C U {d}, < + i mod * = 0, and for all c" / c + 1 mod t, 
= and n", = n' c „. Since j > 0, c + (t — j) mod t ^ c. Thus, the slot by which s± 
differs from s 2 will remain unchanged, and = add(s 2 ,c, d). 

By similar reasoning, if s 2 = swap(si, c, d, d') for some d', s[ = swap(s 2 , c, d, d'). Thus, either 
way, s[ Tl 2 /^ s 2 . (3 that maps the one state of Supp(z/i) to the one state of Supp(z/ 2 ) shows 

that v\ C(TZ 2 ^, 0) v 2 since | In [i\{s'^) — In ^ 2 (add(s' l5 c, d))\ = | In 1 — In 1| = < S. 

• Subcase: a G Q. In this case, we prove that such a 5 exists using 5 = 2e. That is, we prove 
that v\ £(7£^~ 2e , 2e) v 2 . In this case, si has the form (08, (B[, . . . , B' t ), d, y', r', k'). v\ is 
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such that 



^((16, (B' Q , . . . , g t _ x ), K, . . . n'^), c', a, r", K a )) = Pr 



and ^i(si) = for all other states s' x . v 2 is either such that 
i/ 2 ((16, . . . , B' c W {d}, . . . BU), K, ...< + !,... n^), c', a, r", « a )) 



Pr 



Ia M+J 5; W {d} 



^=0 



or 



z/ 2 ((16, <B£, . . . , ^ W {4 - {d'}, . . . B'^), (n' , . . . c', a, r", n a )) 



Pr 



*a l+J B' e w {d} - {d'} 



^=0 



for some d' and ^i(s 2 ) = for all other states s' 2 . Let ,6 denote which of 1+)^ = q -B^ l±l {d} and 
l+|^o 5; tt) {4 - {d'} it is. Either way y^ = Q Bg and B differ by at most two elements Since n a 
has e-differential privacy, we know that for any r", 



Pr 



^=0 / 



<exp(2e)*Pr [re (B) = r"] 



Thus, 



z/i({16, (Sq, . . . , J B^ 1 ),c',a,r",K a )) 

< exp(2e) * z/ 2 ((16, {B' , . . . , B' C U {d}, . . . B' t _ t ), c', a, r", «»)) (120) 

and 

z^i((16, (Sq, . . . ,-B^_ 1 ),c',a,r // ,K )) 

< exp(2e) * ^((16, (S , . . . , B' c W {4 - {d'}, . . . B{_i)>c, a, r", « a )) (121) 

Similarly, 

*/ 2 ((16, (B' ,...,B' C U {4, . . . B^.c', a, r", ««,)) 

< exp(2e) * ^((16, (B' , . . . , 5^), c', a, r", « a )) (122) 

and 

z/ 2 ((16, (-Bq, . . . , i^W {d} - {d'}, . . . BU),^, a, r", « a )) 

< exp(2e) * ^((16, (B' , B[_ x )^, a, r", K a )) (123) 
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isInLif tedRelation^^, R, 5, v\, 1/2) 

Vl ■= {} 

:= {} 
E:={} 

for all xi G S± 
if v\(x{) > 0, 
add xi to Vl 
for all %2 G S± 
if ^2(^2) > 0, 
add X2 to Vr 
for all xi G V\_ 
for all %2 G Vr 

if(xiRx2 and | \nv\(x\) — In V2 (x^) | < 5) 
add edge (xi,X2) to E 
return Hopcrof tKarpHasPerf ectMatching((VL, Vr, E)) 

Figure 5: Algorithm for checking 5-approximate lifting of relations. 

To show that v\ C(TZ 2 J^~ 2e , 2e) u^, we use a function (3. In the case where S2 = add(si, c, d), (3 
maps each state s[ of Supp( / ui) to add(s^, c, d). To show that j3 is a bijection from Supp(/^i) 
to Supp(/i2) note that add(-,c, d) is a bijection and that Lines 11201 and 11221 imply that is 
in Supp(^i) iff add(s' 1 , c, d) is in Supp^)- 

In the case where S2 = swap(si,c,d,d'), j3 maps each s[ to swap(s' 1; c, d, d'). To show that 
(3 is a bijection from Supp(//i) to Supp(/Z2) note that swap(-, c, d, d') is a bijection and that 
Lines [T2~T1 and [1231 imply that is in Supp(/ii) iff swap(s' 1 , c, d, d') is in Supp(^2)- 

Since Tl 2 }^ ^ 6 = T^^^i f° r au r "> s 'i ^/d"^ P(s[). Furthermore, for all s' x in Supp(//i), 
|ln//i(s'J - ln^^^i))! < e < 2e < 5 from' Lines LLM LHH LESS and[[23 

This completes the proof of the lemma. 

Since Tt 2 /^ 6 covers s and d for all states s and data points d of the automaton M ex i, Lemma [2] 
and Theorem [2] together prove that the automaton has (2t * e)-differential noninterference. 

G The isInLif tedRelation Algorithm 

The reduction used by isInLif tedRelation is shown in Figure [SJ First the algorithm constructs 
the bipartite graph for the reduction and then uses the Hopcroft-Karp algorithm |HK73] . This 
algorithm returns if and only if there exists a perfect matching M for the graph. A perfect matching 
M for a bipartite graph (VL, Vr, E) is a subset of E such that for every vertex v G V = Vj_ U Vr is 
incident to exactly one edge in M. 

(Since Supp(z/i) and Supp(i/2) might not be disjoint, but V\_ and Vr must be disjoint, we should 
tag the states xi and X2 differently before adding them to the sets. However, for readability, we do 
not explicitly do this tagging.) 

Proposition 16. For all sets S, relations R over S, non-negative reals 5, and distributions v\ and 
V2 over S, isInLif tedRelation^, R, 5, v\, V2) returns true iff v\ C(R, 5) U2- 
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Proof. By the correctness of the Hopcroft-Karp algorithm, Hopcrof tKarpHasPerf ectMatching 
(and, thus, isInLif tedRelation) will only return true if there exists a perfect matching M for 
the graph. 

To prove only-if direction, assume that such an M exists. Given a perfect matching M of 
bipartite graph, for every x\ G V\_ there exists a unique edge e £ E such that there exists a x 2 G Vr 
such that e = (xi,X2). For each such x±, denote the unique X2 paired with it by this edge as 
(3m{x\). Pm is a function from Vj_ to Vr since for every x\ G V\_, there exists exactly one such edge 
and, thus, exactly one such X2, which must be in Vr since the graph is bipartite. Furthermore, (3m 
is a bijection since every X2 in Vr must be incident to exactly one edge in the perfect matching M. 

Since V\_ = Supp(i^i) and Vr = Supp^), (3m is a bijection from Supp(^i) to Supp^). Since x\ 
and /3m(%i) are connected by an edge, x\ R (3m(%i) and | lnz^i(xi) — In iaj ((3m | < 8. Thus, the 
bijection (3m is such that for all x\ G Supp(z^i), x\ R (3m{xi) and |lnz^i(xi) — In V2((3m(%i))\ < 8. 
This implies that v\ C(R,8) V2- 

To prove the if-direction, assume that v\ £(R,<5) V2- Then there exists a bijection (3 from 
Supp(Vi) to Supp(zv 2 ) such that x\ R (3(x\) and | ln^i(xi) — In V2(f3{xi))\ < 8. Let Mg be the set 
such that (x\,X2) G Mp iff = X2- Mp is a subset of E since x\ G Supp(z^i), f3{x\) G Supp(z^2), 

xi R (3{x\), and |lnz/i(xi) — In V2 (/3(xi)) | < 5 together imply that (xi,/3(xi)) is in E. Mg is a 
perfect matching for the graph since (3 is a bijection from Vj_ = Supp(^i) to Vr = Supp^). □ 

Proposition 17. isInLif tedRelation runs in 0(\S\ 2,5 ) time. 

Proof. Given that we know that we never will attempt to add a duplicate element to any of the 
sets Vj_, Vr, nor E, all the set operations may be done in constant time. Thus, constructing the 
graph for the reduction operates in OdS*! 2 ) time. The Hopcroft-Karp perfect matching algorithm 
operates in 0(y/v * e) time where v is the number of vertices and e, the number of edges. That is 
lower than Od^l 2 ' 5 ) since e < v 2 and v = \V\_\ + |Vr| < 2 * |5|. Thus, the whole algorithm runs in 
0(\S\ 2 - 5 ) time. □ 

H Proofs for the Checking Algorithm 

Proof of Lemma [3l The Soundness of isUnwindFam. Here rel represents the relation family 
1Z such that 7Z € is equal to rel[[e/<5J] for e such that < e < t * 5. If such a family is an unwinding 
family for transition system, then it is also one for the transition system with all the hidden states 
have been converted to the same one. 

We prove a stronger fact that implies that 1Z is an (i * <5)-unwinding family for the converted 
transition system. Namely, we show that the algorithm will only return true if for all e from 
[0,t5], for all x\ and X2 in S± such that (x\,X2) G rel[|_e/£J], for all a in I U R, there exists 
v\ such that x\^v\ iff there exists V2 such that X2=§ > z^2) and when they do exist, either (1) 
vi £(rel[[e/«5J] -0,0) v 2 or (2) v x £(rel[Le/5J -8], 8) v 2 . Condition (1) is satisfied if for all 
(x'^x^) G rel[[e/(5J], f\{x'^) = v(x' 2 \ Condition (2) is satisfied if for all (x'^x^) G rel[[e/5J — 8], 
| lnfi(x^) — ln^(x 2 )| < 8. 

The algorithm will only return true if none of the preceding return statements return false. 
Firstly, it must be the case that |rel| = t + 1. Secondly, the outer most for loop must finish 
executing without any of its return statements being reached. This will only happen if for all 
values of i from the length of the array. For each such value, the algorithm examines the relation 
rel[z], which is the relation used for all values of e in [0,t8] such that [e/8\ is equal to i. Thus, 
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by considering each value of i, the algorithm examines the intervals [0,6), [5,25), and so on up to 
[(t — 1)5, t5) and finally the point at t5. Thus, it examines the whole range [0, t5] as required by 
the above condition. 

Each of these examinations consists of looking at every pair of (x\,x-2) in the relation rel[i], 
and every action a in I L) O. For each such action a and pair, the algorithm first returns false if it 
is not the case that x\ =4> v\ iff X2 =£- i>2 for some v\ and 1/2 since T[xi][a] is equal to nil only in the 
case where x\ =>• v\ for no v\ (and likewise for xi). 

If false was not returned, the algorithm checks if it was because v\ and vi both exist. If this is 
not the case, the examination finishes as nothing more must be shown for this state-action pair. 

In the case where v\ and vi do exist, we know that x\ and xi are actual states and _L since 
T[_L][a] = nil for all a. The examination then continues with the algorithm computing the values 
of v\ and V2 such that x\ =4> v\ and xi =4> as described above, which is well defined since x\ and 
X2 are actual states. 

Next, it checks if v\ £(rel[i],0) Vi- isInLif tedRelation^^, rel[i], 0, u\, v<i) will return true 
iff Condition (1) is satisfied. If Condition (1) is satisfied, the examination is complete and algorithm 
does not return false on this execution of the loop's body. 

If Condition (1) is not satisfied, then algorithm next checks to see if Condition (2) holds. For 
our restricted set of relation families, Condition (2) cannot hold if i is and Condition (1) does not 
hold. Thus, the next if statement. It uses isInLif tedRelation(S , j_, relfz — 1], 5, v\, 1/2) to check 
if Condition (2) holds. If any pair is not, the algorithm returns false. If Condition (2) is satisfied, 
the examination is complete, and algorithm does not return false and this execution of the loop. 

Thus, each execution of the loop will only complete without returning false if either Condition 
(1) or Condition (2) holds. As the loop checks all the needed combinations of states and actions, 
the algorithm will only return true if the stronger fact that implements 1Z is an unwinding relation 
is true. 

Proof of Lemma[4t The Running Time of isUnwindFam. The conversion of all hidden actions 
to the same one runs in 0(|iT| * \S\). 

The outer most loop runs over the whole length of rel. The next loop is over every pair in 
rel[i] where rel[i] is a binary relation over states. Thus, there are at most IS*) 2 pairs in rel[i]. 
The next loop is over every action. Thus, the body of this loop will be executed 0(t * \S\ 2 * \A\) 
times. 

This body consists of four parts. The first is a simple conditional taking constant time. The 
second computes v\ and V2- This takes 0(|-ff| * 15*1 + IS"! 3 ) time. Since the conversion of all hidden 
actions to the same one takes \H\ = 1, this is Od^l 3 ). The third is a calls isInLif tedRelation, 
which takes Od^l 2 ' 5 ) time. The forth is a conditional and another call to isInLif tedRelation 
on rel[z — 1], which takes 0(|5| 2 ' 5 ) time. Thus, body is 0(|<S| 3 ) time and the whole loop is 
0(t*\A\ * |5| 4 ). 

The algorithm whole algorithm run in 0(|i?| * \S\ + t * \ A\ * \S\ 4 ), which is 0(t * \A\ * |iS| 4 ) since 
\H\ < \A\. 

Proof of Theorem [4) The Soundness of isAHCovered. The algorithm will only return true 
if none of the preceding return statements return false. That is, the outer most for loop must 
finish executing without any of its return statements being reached. This will only happen if for 
every reachable state s and every data point d, either T[s][cZ] = nil or each of the following is true: 
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1. v{±) = and |Rels[s][d]| ^ t = 1; 

2. for all states s' such that s' £ Supp(^), (s,s') G Rels[s][d]; and 

3. isUnwindFam((S, /, O, T), Rels[s] [d], 5, t) returns true 

where s =i v. In the case where T[s] [d] = nil, the trivial relation family that consists of only empty 
relations is a (t * <5)-unwinding family for the automaton. In the case where T[s][d] ^ nil, the 
three conditions above imply Rels[s][d] is a (t * <5)-unwinding family for the automaton by using 
Lemma [3] on the last condition. Either way, there exists a (t * <5)-unwinding family that covers s 
and d. Thus, the body of the loop will return false unless there exists such an unwinding family. 

As the algorithm checks every reachable s for every d, the loop will not terminate without 
returning false unless the conditions of Theorem [2] holds. Thus, the algorithm only returns true if 
the automaton has (t * ^-differential noninterference. 

Proof of Theorem [5} The Running Time of isAHCovered. Computing the reachable states 
can be done in time 0(|5|). 

The outer most loop executes at most \S\ times. The next loop executes at most \D\ times. In 
the case where T[s][d] 7^ nil, the body takes 0(5 3 ) time to compute u, 0(\S\) for the inner loop, 
and 0(t* \A\ * \S\ 4 ) time for running the isUnwindFam algorithm (LemmaH|). Thus, the body takes 
0(t * \A\ * |5| 4 ) time and the whole algorithm takes 0(t * \D\ * \A\ * \S\ 5 ) time. 
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